Method of displaying two adjacent photos on a split screen digital picture frame

ABSTRACT

A method and apparatus for displaying digital photos on a digital display with a network connection module. The method includes identifying two different photos in a digital photo collection that include a matching photo content, and displaying the two different photos simultaneously in a split screen on the digital display. The two different photos can be matched from related photo clusters. An integrated camera can be used to automatically determine an identity of a frame viewer to determine photos to display, and can capture gesture-based feedback. The photos can be filtered and cropped at the receiver side.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Pat. Application, Serial No.17/084,987, filed on 30 Oct. 2020, which is a continuation-in-part ofU.S. Pat. Application, Serial No. 16/113,366, filed on 27 Aug. 2018,which is a continuation-in-part of U.S. Pat. Application, Serial No.15/824,337, filed on 28 Nov. 2017, which is a continuation-in-part ofU.S. Pat. Application, Serial No. 15/150,623, filed on 10 May 2016,which is a continuation-in-part of U.S. Pat. Application, Serial No.14/949,353, filed on 23 Nov. 2015, which is a continuation-in-part ofU.S. Pat. Application, Serial No. 14/455,297, filed on 08 Aug. 2014,which is a continuation-in-part of each of: U.S. Pat. Application,Serial No. 14/051,071, filed on 10 Oct. 2013, and U.S. Pat. Application,Serial No. 14/051,089, filed on 10 Oct. 2013. The co-pending parentapplication(s) are hereby incorporated by reference herein in theirentirety and are made a part hereof, including but not limited to thoseportions which specifically appear hereinafter.

FIELD OF THE INVENTION

This invention relates generally to a digital picture frame and, moreparticularly, to a digital picture frame that automatically providespersonalized photo viewing, such as with two photos on a split screen,in automated combination with a proximate or remote electronic deviceand/or a Mobile Positional Social Media (MPSM) account.

BACKGROUND OF THE INVENTION

Existing digital picture frames synchronize with popular social mediaplatforms, such as Facebook® and Instagram®, using, among othertechniques, a WiFi® interconnection. However, multiple steps may berequired for such synchronization. These manual steps can be difficultto do if one wants to change to a different social media account or todirectly retrieve from a picture capture or storage device, such aswhile entertaining guests. There is a continuing need for improvementsfor implementing displays such as home digital picture frames.

SUMMARY OF THE INVENTION

The present invention provides a digital picture frame thatautomatically detects viewers and/or viewers’ electronic devices, andautomatically displays photos or videos relevant to the detectedviewers. The relevant photos or videos are obtained upon detection fromthe electronic device and/or social media accounts of the detectedviewers. The photos or videos can also be automatically obtained fromdevices, frames, stored photo collections, or social media accounts ofone or more community members of the detected viewer(s), such as photostaken and posted by a friend or relative of the detected viewer. Thedevice and method of this invention provides for improved and updatedphoto display based upon current photos and/or social media updates, andthus relevant to the viewer. The device of this invention can be used asan MPSM device to display any MPSM information, as desired, includingbut not limited to MPSM information obtained according to methodsdescribed herein.

A general object of the invention can be attained, at least in part,through a digital picture frame including a digital display mountedwithin a frame, a camera connected to the frame, a network connectionmodule adapted to connect to an electronic device, photo collection,and/or social media account of a viewer viewing the digital pictureframe and to receive photos stored within the electronic device and/orthe social media account, and an automated display module adapted toautomatically change photos displayed on the digital display to photosfrom the electronic device and/or social media account of the viewer,such as upon automatic detection with the camera of the viewer.

Embodiments of the invention include a method of displaying digitalphotos on a digital display with a network connection module,comprising: identifying two different photos in a digital photocollection that include a matching photo content; and displaying the twodifferent photos simultaneously in a split screen (digitally created) onthe digital display. The split screen desirably includes two equalsections of the digital display. A first of the two different photos isin, or forms, a first of the two equal sections, and a second of the twodifferent photos is in, or forms, a second of the two equal sections.

The matching photo content can be a same person or a same combination ofpersons. The two different photos can be photographically captured intwo different calendar years, wherein the two different calendar yearsare separated by a time period, such as more than one additionalcalendar year. The matching photo content can be a same person and eachof the two different photos further include one of two different secondpersons, for example, wherein the different second persons are relatedfamily members (e.g., 1: Jane with mon and 2: Jane with dad). The twodifferent photos can be automatically downloaded from one or moredigital photo collections of one person or two different second persons.

The invention further includes a method and device for displaying photoson a digital picture frame that includes: automatically clusteringphotos of one or more digital photo collections for display to theviewer, and automatically displaying on the digital display the photosfrom the digital photo collection as a function of the clustering. Theclusters are groups of photos to form photo collections of relatedcontent, such as selected from people, things, places, activities, orcombinations thereof, and displaying the photo collections. The photosare desirably clustered by a photo content, which can be identified orconfirmed by photo metadata. Clustering of photos with a preselectedphotographed person can be performed by automatically identifying andclustering the photographed person by facial features. Clusteringaccording to this invention is particularly useful for photos taken overa multi-year timeframe, and grouping photos by related photo contentacross the timeframe.

Embodiments of this invention further include automatically clusteringdigital photos of the digital photo collection according to a content,and automatically selecting the two different photos for the splitscreen display as a function of the clustering. The two different photoscan be selected from a same or two different photo clusters. In someembodiments, the invention includes scanning digital photos for content,automatically creating a plurality of photo clusters of the digitalphotos for display from the digital photo collection, wherein each ofthe photo clusters is based upon a different cluster content, andselecting the two different photos from at least one of the photoclusters.

Embodiments of this invention include downloading a first interactionapplication to a first electronic device of a first user, the firstelectronic device including a first digital photo collection, anddownloading a second interaction application to a second electronicdevice of a user community member, the second electronic deviceincluding a second digital photo collection of the user communitymember. Each interaction application automatically clusters digitalphotos of the corresponding digital photo collection according tocontent. Selected or all photo clusters are provided to the device fordisplay. The two different photos can come from the first and/or thesecond selected digital photo clusters. For example, a first of the twodifferent photos can be from the first selected digital photo and asecond of the two different photos can be from the second selecteddigital photo cluster.

In embodiments of this invention, the clustering includes preparing aplurality of sub-clusters, each for a common detected content in thephotos selected from things, places, activities, or combinationsthereof, wherein each of the plurality of sub-clusters is formed of thecommon detected content at a similar time and/or age. Relatedsub-clusters of the common detected content are linked together to forma cluster of related common detected content for display as a functionof one or more predetermined features having limited change over atimeframe. The first of the two different photos for the split screencan be selected from a first of the sub-clusters and a second of the twodifferent photos from the second of the sub-clusters. For clustering,the method desirably operates by extracting photo features of the commondetected content within each of the sub-clusters of photos to formcluster representations, wherein the photo features are predetermined ashaving the limited change over a timeframe, comparing the clusterrepresentations between sub-clusters to determine the relatedsub-clusters, and linking or grouping the related sub-clusters fordisplay as a function of matching cluster representations.

The invention further includes automatically clustering from the digitalphoto collection, via automated facial recognition, photos over amulti-year timeframe of a person to obtain clustered photos of theperson. The clustering includes: preparing a plurality of sub-clusterseach including digital photos of the person at a similar time and/orage; extracting photo facial features within each of the plurality ofsub-clusters that comprise a predetermined limited change over thetimeframe to form a cluster representation of the person for each of theplurality of sub-clusters; comparing cluster representations between theplurality of sub-clusters to determine related sub-clusters of theperson over the timeframe as a function of the cluster representations;and linking or grouping related sub-clusters to form the clusteredphotos of the person for display. A first of the two different photoscan be selected from a first of the sub-clusters and a second of the twodifferent photos from the second of the sub-clusters.

The invention includes a method of displaying photos on electronicdevices with a network connection, including: automatically obtaining adigital photo from a first electronic device; automatically identifyinga content of the digital photo; automatically identifying a targetdisplay electronic device as a function of the content; andautomatically routing the digital photo over the network connection tothe target display electronic device for display. The first electronicdevice is controlled by a first community member and the target displaydevice, such as a mobile device or a digital picture frame, can becontrolled by a second community member. The target display electronicdevice desirably includes predetermined content constraints, and thedigital photo is routed and/or displayed if the photo meets the contentconstraints.

The invention further includes a method of displaying photos onelectronic devices with a network connection via: automaticallyobtaining digital photos on a first electronic device; automaticallyidentifying a content of each of the digital photos; automaticallymatching the content of a first of the digital photos to a predeterminedcontent constraint of a second electronic device; and automaticallyrouting the first of the digital photos to the second electronic deviceover the network connection for display on the second electronic device.The method can further include automatically processing the digitalphotos by at least one of image enhancement, content filtering, and/orrepetition filtering. The content filtering can include identifying theplace, person, and/or activity, etc. of the photo, which can then becompared to preferred content (e.g., a stored metadata identificationset) of the potential target/receiving frames.

In presently preferred embodiments of this invention, where the contentof the first of the digital photos matches a predetermined contentconstraint of a plurality of community members, the first of the digitalphotos is automatically routed to electronic devices of each of thematching community members. The invention further includes a method ofdisplaying photos on electronic devices with a network connection,including: automatically obtaining digital photos on a first electronicdevice; automatically identifying a content of each of the digitalphotos; automatically matching the content of each of a plurality of thedigital photos to a predetermined content constraint of a correspondingelectronic device of one or more community members; and automaticallyrouting each of the plurality of digital photos to the correspondingelectronic device as a function of the matching for display on thecorresponding electronic device.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes: automatically determining with the camera anidentity of a viewer of the digital picture frame; automaticallydetermining a viewer profile from the identity of the viewer; andautomatically changing photos shown on the digital display as a functionof the viewer profile upon detection of the viewer with the camera. Themethod can include automatically augmenting a photo display on thedigital display as a function of photo metadata and the viewer profile.In embodiments of this invention, the viewer profile is automaticallylearned and uploaded via an electronic device and/or a social mediaaccount of the viewer.

The invention further comprehends a method of displaying photos on adigital picture frame including a digital display mounted within aframe, a camera connected to the frame, and a network connection module.The method includes automatically determining with the camera anidentity of a viewer of the digital picture frame, automaticallyconnecting to a picture capture or storage electronic device of theviewer over a network, such as a WiFi network, and automaticallydisplaying photos shown on the digital display obtained from theelectronic device over the network connection.

The device and method of this invention can automatically display aslideshow of photos relevant to one or more display viewers as afunction of photo context selected from time taken, photo location,and/or photo content. The device can automatically access, using thenetwork module, at least one electronic device, digital photocollection, and/or social media account of: the viewer and/or at leastone social media community member of the viewer. Photos from the one ormore electronic devices, photo collections, or social media accounts aredesirably automatically loaded and displayed. In embodiments of thisinvention, obtained photos are automatically sequenced on the digitaldisplay as a function of profiling traits selected from chronologicalorder, photo location, photo activity, and/or community member.

The device and method can detect and recognize two or more different andsimultaneous, or otherwise present, persons viewing the digital pictureframe, and automatically change the photos displayed on the digitaldisplay to photos uploaded from an electronic device and/or a socialmedia account of each of two or more persons viewing the digital pictureframe. The photos from the electronic device or social media account ofeach of the two or more persons can include shared activity photos fromactivities shared by the two or more persons. The invention includesembodiments where a server computer automatically determines sharedactivity photos from the electronic device and/or social media accountof the each of the two or more persons as a function of contextinformation automatically associated with the shared activity photos bythe same or different server computer. The context information of eachof the shared activity photos can include, but is not limited to, aphoto location, a photo activity, and/or a present community member.

The invention still further comprehends a display device with one ormore of: automatic detection for configuration, namely theself-straightening or self-positioning on a vertical (e.g., wall) and/ora horizontal (e.g., table) platform; the automatic detection of viewingconditions, namely, based on the distance of the viewer, enlarging orcontracting the picture; based on the lighting, increasing or decreasingthe lighting intensity. The invention also comprehends a display devicethat: via face recognition, automatically detects the viewer and targetsthe photos for her or him based on their potentially learned profile;via the automatic identification and tagging of the location, activity,and community member involvement on a per photo or video, indicating thewith whom, where, when and what was being done when the picture wastaken as well as any other associated metadata; and/or providing a“story telling” capability. By storytelling, chronological stories,optionally simultaneously displayed on a split screen, are grouped by:purely time, namely in sequential or reverse chronological ordering;location, namely a traversal of sites on a location based trip;activity, namely in chronological ordering of a given or similar set ofactivities; community member involvement, namely a pictorial interactionwith community members, potentially segmented by particular communitymember or members; or any other profiling traits of a recognized userthat can be used to cluster or segment photos or videos for automaticstory telling.

The invention includes a method of displaying photos on a digitalpicture frame and including: automatically determining with the cameraan identity of a viewer of the digital picture frame; automaticallyconnecting to one or more digital photo collections over a network; andautomatically displaying on the digital display photos obtained from thedigital photo collection over the network connection. The digital photocollection(s) can be associated with a second digital picture frameand/or an electronic device of a community member of the viewer. Thedigital picture frame has access to the digital photo collection(s) anda server computer automatically sends the photos to the frame throughthe network connection module. The server or frame desirablyautomatically crops and aligns the photos to fit the digital display,such as by automatically scaling and realigning the photos to an aspectratio of the digital display.

Additional automated processing can include image enhancement, contentfiltering, or repetition filtering. The photos can be filtered by imagequality and content quality. The digital picture frame can include acontent restriction filter module including adjustable contentrestraints controlled by a frame administrator, for implementing contentrestrictions of the viewer obtained from a viewer profile over thenetwork. The frame administrator and/or the viewer can have or setpre-established filtering parameters selected from image quality and/orcontent quality, for use in filtering the digital photo collection.

In embodiments of this invention, the device and method includeautomatically detecting via the camera a physical response to one of thedisplayed photos, and automatically assigning an interest level grade tothe one of the displayed photos using the physical response. Thephysical response can be a body movement or a facial expression. Theinterest level grade can be stored in a viewer profile for the viewer,and used for displaying photos for the viewer as a function of theinterest level grade. The physical response, positive or negative, canbe used to identify and suggest other related photos to the viewer, suchas for a replacement photo related to the same subject matter.

In embodiments of this invention, the device and method includeautomated ranking of the photos to be circulated on the frame, such asfor automatically determining an order of presentation, a number ofdisplay repetitions, and/or a time period of display for each of thephotos as function of the ranking. The ranking can be based at least inpart on a physical feedback response of the viewer to one or more of thephotos, or based upon a picture quality and/or content. The photos canbe clustered according to the rankings into groups for display orsharing.

The invention can include the incorporation or creation of usercommunities and sub-communities, with such communities andsub-communities sharing information. The digital picture frame ofembodiments of this invention operates in concert with a method, system,and apparatus, such as embodied in an MPSM or other softwareapplication, that automatically determines and shares a location and/oran activity of a user. The application learns user activity over time,with the learning based upon user locations and/or context. The presentinvention generally provides methods and applications for an MPSM thatautomatically understands and informs the “who, what, when, where,and/or how” of a user and the user’s community. For example, who are theuser and their community with, what are the user and their communitydoing, where geographically are the user and their community, when are,and historically when were, the user and their community doing this,and/or how can users’ behaviors be modified? The method includesautomatic tagging of photos taken by the users with correspondingcontext data (e.g., location, activity, people present) learned by thesystem for use in determining photos to display on the digital pictureframe of this invention as discussed above.

Other objects and advantages will be apparent to those skilled in theart from the following detailed description taken in conjunction withthe appended claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIG. 2 illustrates a digital picture frame in network connection to aserver and mobile devices, according to one embodiment of thisinvention.

FIGS. 3 and 4 illustrate a photo clustering according to one embodimentof this invention.

FIGS. 5 and 6 each illustrate a digital picture frame in networkconnection to a server and mobile devices, according to embodiments ofthis invention.

FIG. 7 shows a representative area of a user according to an MPSM methodof one embodiment of this invention.

FIG. 8 illustrates geofences surrounding a current reading and itsimmediate neighbors according to one embodiment of this invention.

FIG. 9 illustrates the determination of a location via intersectingcircles according to one embodiment of this invention.

FIG. 10 illustrates the processing flow employed to identify an arrivalaccording to one embodiment of this invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a digital picture frame that interactswith either a picture capture or storage device, referred to herein asan electronic device, running a digital picture frame application and/orsocial media to automatically display digital photographs from theelectronic device and/or a social media account of one or more personsviewing the picture frame. As used herein, the terms “picture”,“photograph”, or “photo” refer to or include digital still photographsor digital video, including a combination of few-second animationseither or both before and after a still frame, unless otherwise noted.The photos can be obtained by social media users and stored in theusers’ social media accounts. When a social media user views the digitalpicture frame, the digital picture frame automatically recognizes thatuser and automatically loads photos relevant to that viewer from her orhis electronic device and/or social media account and/or from at leastone electronic device and/or social media account of her or his socialmedia community member(s). For example, relevant photos from a communitymember’s social media account may include the viewer or be from a sharedactivity that the viewer attended with the community member. Automatedsocial media activity learning methods, as described herein, are used toautomatically determine relevancy of photos to one or more communitymembers. Community members may choose to make photos shared or privateto avoid personal photos being shared.

FIG. 1 illustrates a digital picture frame 20 according to oneembodiment of this invention. The digital picture frame 20 includes adigital display 22 mounted within an edge frame 24. The digital display22 can incorporate any suitable screen, such as, without limitation, anLED screen or a touch screen, as are known and commercially available. Acamera 25 is connected to and integrated in the edge frame 24. Thepicture frame 20 includes a network connection module that connects overa wired and/or wireless network to a social media server computer 26 andcan automatically load photos stored within one or more social mediaaccounts accessed through the server 26. The picture frame 20 furtherincludes an automated display module in combination with the networkconnection module and adapted to automatically change photos displayedon the digital display 22 to photos from an electronic device and/or asocial media account of the viewer and/or one or more community members,such as upon automatic detection with the camera of the viewer. Thedigital picture frame 20 desirably includes a microphone 27 incombination with the automated display module and/or network connectionmodule, such as for receiving spoken instructions from a viewer tochange the displayed photos.

The digital picture frame 20 further includes a leveling device 28connected to or otherwise integrated with the frame 24. The levelingdevice 28 automatically detects when the frame is not level. Properpositioning of the digital picture frame is important to achieving usersatisfaction. Auto detection of the positioning in comparison to theangle of the displayed photograph illustrated can be achieved using anyof the image alignment or realignment techniques known in the art. Inembodiments of this invention, the leveling device 28 can transmit ordisplay corrective measures for physical correction.

In another embodiment of the invention, the digital picture frame 20 isconfigured with a leveling device that includes one or more motors 23,such as paired attached with a corresponding wall bracket via gearing.The motors 23 can be incorporated in a wall hanging mechanism as shownin FIG. 1 , or otherwise implemented with respect to a frame stand. Upondetection of an unleveled viewing as determined by either the levelingdevice 28 or any other mechanism, the dual motors 23 activate andautomatically mechanically operate as needed to level the digitalpicture frame 20, such as a vertical adjustment on wall mountinghardware, via the adjustable hanging mechanism or bracket.

The digital picture frame 20 includes an automated display adjustmentdevice in detection combination with the camera 25. The automateddisplay adjustment device automatically detects frame environment and/orviewer position using the camera 25 and automatically adjusts settingsof the digital display 22. Augmented with the camera 25, the digitalpicture frame 20 supports a host of optional, additional capabilities.Using conventional, known in the art, image processing techniques,enhancement of the picture quality can be automatically made. Forexample, due to an automatically detected viewing angle or environmentalbrightness, picture lighting intensity and contrast can be altered. Thatis, depending on conditions, the brightness and contrast can beautomatically modified to provide better viewing. Additionally,depending on the distance and angle of the viewer, the size of the imagecan be enlarged, possibly focusing on the center or content rich area ofthe picture. Similarly, a close viewer may wish to have the photographor video in full scope, or maybe, even side-by-side with one or moreother, possibly related pictures or videos.

In one embodiment, as shown in FIG. 1 , the digital picture frame 20includes the capability to ‘daisy chain’ one or more additional frames20′. That is, all participating frames can be connected so as to portraya single story, either chronologically or thematically, allow for twoviewers, and/or provide a composite view of a single image. The frame 20can include a network connection 10, such as an audio-video outletand/or network outlet, to connect to a corresponding inlet 12 of thesecond frame 20′. The network connection between frames can be by wiredconnection (e.g., HDMI cord or Ethernet cable) or wireless communication(e.g., Bluetooth or WiFi) The frame 20 can optionally include a poweroutput, such as outlet 14 to receive a power cord 16 of the second frame20′, so as to require only a single power source.

An embodiment of a digital picture frame 20 further includes multiplepower options. In addition to conventional outlet and battery options,the frame can include at least one photoelectric cell. Inductive, orwireless, charging can also be incorporated.

The digital picture frame 20 includes a facial recognition module withany suitable facial recognition software, in combination with the camera25 and automated display module. Face recognition software can bedeployed to automatically detect the identity of the viewer(s). It isalso within the scope of this invention to identify the viewer via thelocation or proximity of their personal electronic devices such as butnot limited to their mobile phones or tablets. By identifying theviewer(s) the automated display module can then specially target, namelyselect, prioritize, and/or order the photos or videos displayed for theidentified viewer.

Identifying the viewer enables a variety of novel applications. Inembodiments of this invention, a viewer analysis module is combined witha facial recognition module to determine, for example, one of mood orhealth of the viewer upon detection of the viewer with the facialrecognition module. The automated display module can augment a photodisplay on the digital display as a function of the determined mood orheath of the viewer, such as by showing photos to increase the viewer’smood.

Longitudinal analysis of the viewer is possible. Such analysis supports,for example, health commentary. By analyzing the viewer’s face, commentssuch as, but not limited to, “you seem to have gained weight”, “you looktired”, or “you seem pale; do you feel well?” can be announced to theviewer. Similarly, “mirror, mirror on the wall” remarks such as “youlook good today” can be stated, potentially elevating the viewer’s mood.Displaying identified mood enhancing photographs, such as, for example,ones that include far away loved ones can likewise be shown to furtherelevate the viewer’s mood. Identification of mood enhancing photographscan be automatically accomplished, for example, via the recognition of a“smiling face” or spoken words of the viewer when the picture waspreviously shown, and tagging of the photo.

Viewer identification also enables photo selection personalization. Aviewer personalization module can be operatively combined with thefacial recognition module, whereby the viewer personalization moduleautomatically identifies preferred photos or restricted photos as afunction of viewer information upon detection of the viewer with thefacial recognition module. Photos favored by the viewer (e.g.,grandchildren for the grandparent) can be selected, and converselyphotos containing, for example, ex-spouses or in-laws can be repressed.Likewise, identifying viewer characteristics, such as age, enables thelimiting of photos displayed. For example, children might not be shownphotos containing potentially inappropriate content, such as containingnudity, or particular activities or individuals, such as relatives thathave recently passed away. Thus, it is within the scope of thisinvention to limit or specifically select photographs to be displayedbased on relationships of the viewer with individuals in thephotographs, such as ex-spouses; characteristics of individuals shown inthe photographs, such as babies; characteristics of the viewer, such asage; activities illustrated in the photographs, such as smoking;locations displayed in photographs both physically described (e.g.,Chicago) or relatively described (e.g., viewer’s place of residence); orbased on any characteristics that can be determined via image analysisor photograph metadata included. The filtration or selection ofphotographs based on criteria is determined by settings that can be set.

Additional specialty settings are included. Such setting place the framein nightlight mode where only a dim glow is illustrated; photo recordmode to simply record all in the line of vision; video chat mode withother devices; and “stranger in the house” mode where only limited orstock photos are shown to prevent the potential identification ofresidents.

The digital picture frame 20 can likewise be used as a viewer assistant.Multiple ailments or conditions, such as but not limited to Alzheimer’s,strokes, and dementia, affect memory. Thus, the digital picture frame 20can be used to remind the viewer of the identity of family members andfriends as well as of occasions, locations, activities, etc., ofrelevance as the photos displayed are labeled both by content and/or bymetadata.

As previously discussed, the network connection module connects over anetwork connection to the social media server computer 26. It canlikewise connect either directly or indirectly to the mobile device 29′of any community member or interested party having an appropriatelyconfigured device, such as digital picture frame application specificsoftware. The connection need not be within a geographic proximity, andit is likewise within the scope of this invention that a geographicallyremote connection is supported via networks such as but not limited toas the Internet.

The social media server computer 26 can be any one or more computersthat implement any suitable social media platform, such as the MPSMplatform and automated learning steps described herein. The servercomputer 26 can obtain or have access to photos and learn user oractivity information associated with each photo from, for example,mobile devices 29 of a plurality of MPSM users or community members,such as according to methods described herein. In embodiments of thisinvention, the automated display module augments a photo display on thedigital display 22 as a function of photo metadata from the social mediaaccount stored on the server computer 26. The photos are automaticallydownloaded from the server 26 to the digital frame 20 upon, for example,automated detection of a corresponding viewer(s).

In embodiments of this invention, the photos can be changed according tothe learned bias of a viewer, expressed and stored as part of a userprofile at the server computer 26 and/or the digital picture frame 20.The picture frame 20, such as via the automated display module, includesor obtains the viewer profile of the viewer upon detection of the viewervia the camera. By establishing a viewing profile for each known orexpected viewer, e.g., if a viewer is or has a community member,individual or group user viewing preferences can be maintained. Aprofile can be established using any one or more of the many knownprofile techniques known in the art, including but not limited to,asking the viewer, learning based on previous user selections, and/orcorrelations with previously automatically learned, via MPSM methodsherein, locations, activities, and/or community member interactions.

In a preferred embodiment, viewer profiles are automatically learnedaccording to methods described herein, and uploaded via an electronicdevice and/or the social media account of a detected viewer and/or heror his community members. The automated display module desirablydisplays a slideshow of photos for a detected viewer that is uploadedfrom: an electronic device of the viewer, and/or a social media accountof the viewer or one or more social media community member of theviewer.

In the digital picture frame embodiments of this invention pictures orvideos displayed can be augmented with the automatically learned andidentified metadata corresponding to the pictures or videos. That is,the information learned, captured and/or displayed identifies thelocation, activity, and community member involvement on a per photobasis, indicating the with whom, where, when and what was being donewhen the picture or video was taken as well as any other associatedmetadata, such as the camera or video equipment used to capture thephoto or video. Slideshow of photos for a viewer can be assembled andshown in a display order automatically determined as a function of photoprofiling traits such as time (e.g., chronological order), photolocation, photo content (e.g., people or activity shown in the photo),and/or community member presence at the time/location that a photo wastaken.

The invention includes a method of displaying photos on a digitalpicture frame, such as shown in FIG. 1 . The digital picture frame ishung or set on a surface by a user, for explanation purpose, namedSusan. The picture frame automatically determines with the camera whenSusan is viewing, or is the primary viewer of, the picture frame. Thepicture frame automatically determines Susan’s profile from herdetection. In embodiments of this invention, Susan’s viewer profile isloaded from Susan’s electronic device and/or social media account, andmay be supplemented by additional information gathered via the pictureframe itself, such as face recognition, account access information,and/or manual entry (e.g., picture frame display preferences). Theviewer profile is desirably created and continually updatedautomatically according to the automated learning methods describedherein. The viewer profile can include information such as, withoutlimitation, family information, community member information, and/oruser favorites (e.g., favorite locations, activities, colors, flowers,animals, etc.), learned through the social media account.

When the camera and coordinated software determines Susan is viewing thepicture frame, e.g., either directly or merely in the vicinity, thepicture frame automatically displays photos shown on the digital displayas a function of the viewer profile information. The photos displayedare automatically augmented or picked as a function of comparing photometadata and the viewer profile. In embodiments of this invention, thepicture frame automatically displays for Susan a slideshow of photos(still and/or videos) relevant to Susan as a function of photo context.The photo context can be selected from the time the photo was taken(e.g., all photos from July 2015), photo location (e.g., all photostaken in Chicago), and/or photo content (e.g., all photos of vacationsor involving her sister and/or her friend Mary). The photos can beimported primarily from her electronic device and/or social media, andoptionally supplemented from photos on a recordable medium of the frame(e.g., hard drive or inserted flash drive or memory card) or photosobtained from a third party web site.

It is within the scope of this invention to prompt the user, possiblyvia any communication mechanism including but not limited to text andvoice commands, to determine the exact wishes of the user. If thedigital picture frame is additionally equipped with a microphone 27 andcorresponding speech processing software, voice commands can likewise beprocessed using any of the many known speech processing systems. Susancan select a context, such as by spoken instructions, to be display. Asan example, Susan may request all photos related to “college”, relatedto “vacations”, including “family”, or including or relating to aparticular person. The digital picture frame will download relevantphotos for display from one or more electronic devices and/or socialmedia accounts, of Susan’s and/or a community member of Susan, that havecontext matching the request. If Susan asks for pictures of her and hersister, the pictures may include her sister or have been taken in thepresence of both her and her sister, such as a picture of the EiffelTower taken during a trip with her sister to France. The Eiffel Towerphoto is identified by context metadata that identifies that both Susanand her sister were present when the photo was taken. The contextinformation is desirably automatically learned and stored with thepicture in the social media account of Susan, using the learning methodsdescribed herein. Alternatively, the Eiffel Tower photo can be part ofthe sister’s social media account, and because her sister is a communitymember and has given sharing rights, the social media server computerautomatically shares the photo to Susan’s digital picture frame.Desirably, any photo relevant to Susan can be automatically shared fromany of Susan’s community members and automatically shown on the pictureframe display.

In one embodiment of this invention, the digital picture frame accessessocial media or her electronic device without detailed instructions fromSusan and loads and sequences photos displayed on the digital display asa function of profiling traits selected from chronological order, photolocation, photo activity, and/or community member. Having automaticallylearned the location, activity, and community member involvement on aper photo or video basis using the method described herein, a “storytelling” capability is supported. That is, in storytelling,chronological stories, optionally simultaneously displayed on a splitscreen, are grouped by: purely time, namely in sequential chronologicalordering; location, namely a traversal of sites on a location basedtrip; activity, namely in chronological ordering of a given or similarset of activities; community member involvement, namely a pictorialinteraction with community members, potentially segmented by particularcommunity member or members; or any other profiling trait(s) of arecognized user that can be used to cluster or segment photos forautomatic story telling.

In one embodiment of this invention, Susan’s friend Mary joins Susan inviewing the picture frame, either directly or within a detectablevicinity of the picture frame. The picture frame automatically detectsthe additional presence of Mary, automatically detecting her identity,automatically obtains or loads Mary’s profile, and automatically changesthe photos displayed on the digital display to those relevant to bothSusan and Mary. The photos could be obtained from Susan’s social mediaaccount(s) or personal picture capture or storage electronic device butpreferably, the photos are obtained from both Susan’s and Mary’s socialmedia accounts and/or local personal picture capture or storageelectronic devices, e.g., mobile phones 29 and 29′. Desirably the photosfrom each of the two viewers are shared activity photos; from activitiesshared by the two friends. The frame 20 and/or server computer 26automatically determines shared activity photos of the two persons as afunction of the learned context information automatically associatedwith the shared activity photos by the server computer. As more peopleare automatically detected, photos from more accounts or devices can beautomatically added, and the photos automatically organized by contextsuch as photo location, a photo activity, and the present communitymembers. Options can be added on the social media accounts to share ornot share photos with community members or their devices according tothis invention.

In another embodiment, Mary and Susan each simultaneously, or withinsome specified duration of time, look at their respective digitalpicture frame 20 and 20′, such as linked as shown in FIG. 1 or networkconnected when each frame in a different location (e.g., theirrespective residences). Remote frames can communicate, via a network,such as through server 26, and indicate to both Mary and Susan, each attheir respective frames 20 and 20′, their respective presence,potentially indicating when Mary and Susan are both viewing or viewed aphotograph from a preselected set of photographs within the specifiedduration of time. The duration of time can be defined in terms of aspecific time or as within a time interval commencing from when eitherMary or Susan viewed the given photograph from within the preselectedset. The frames 20 and 20′ can automatically or upon instructioncoordinate so that the two are viewing the same photos. Likewise thephotos can be augmented to included shared experiences, and be uploadedfrom both of Mary and Susan’s linked devices 29 and 29′ and/orrespective social media account.

In embodiments of this invention, photographs from digital photocollections are provided to the frame in multiple ways including but notlimited to self-capture, transmission from remote photograph capturingdevices, from other frames, and through server transmission. The frameeither receives photographs taken by other devices or itself takes aphotograph, if so configured. The digital photo collection(s) from whichphotos can be obtained and sent to the frame can be associated with oneor more additional viewers and/or digital picture frames, and/or anyelectronic device of a community member of the viewer.

Independent of the means by which a photograph is obtained, at times,the main artifact within the photograph, such as, for example, the faceof a person in the picture, is not properly aligned. As such, whendisplayed on the frame, either the main artifact is misaligned, orpotentially, is removed all together. To mitigate potentialmisalignment, the frame and method of this invention can include orprovide an automated smart cropping feature for automatically croppingand aligning the photos to fit the digital display of the frame.

For proper viewing, the frame or the server driving the disclosed framecan automatically crop and/or realign images. Without requiring userinteraction, the server executes cropping software that realigns animage for proper display on the frame to ensure that the main artifactwithin the image is automatically scaled and realigned to a given aspectratio (4:3 or 3:4 for example). The artifact may not be a single thingor entity, and, for example, could be a group such as two faces in theforeground close to each other or a tight group of people in a familyshot. An exemplary non-limiting approach identifies the face areas andattempts to align a corresponding area at either center to top-33% ofthe image to be portrayed, depending on the size. Large faces can becentered vertically whereas smaller faces can be aligned in a top 33% ofthe display to show more of the person’s body. Thus, when presented, theimage is esoterically properly displayed.

Embodiments of this invention preferably also include image filtering,either at the frame or server level. Filtering can include removingimages of faulty artifacts, such as, for example, low quality (blurry,etc.) images. Artifacts of unlikely interest can also be filtered. Forexample, photographs of pets containing faces are typically of interest,however animal photographs without heads are typically not of interest,and can be removed. The curator of the photographs, such as the owner orviewer of the frame, can additionally impose other image or contentquality constraints. Content constraints can span people, places,artifacts, and actions; namely anything that can be recognizedautomatically by any image, photograph, or document processingtechniques known in the art.

Embodiments of this invention enable the owner or user of a frame tospecify additional constraints on the photographs displayed. Such as inthe case of the photograph curator, the frame owner too may designatecontent restrictions, for example, pornographic photographs can beprohibited to display. Such constraining rules are defined andincorporated into the filtering approach. Note that it is also withinthe scope of this invention that the constraining rules include but arenot limited to lighting characteristics, scene selections, number ofindividuals displayed, distance or horizon specifications, or any otherphotograph characterization features.

The invention further includes receiver (displayed) side based filteringwhere the filtering of content to display to be defined by not only bythe curator/owner of the photographs but also the owner/user of thedisplaying frame. In the case of multiple owners/users of a displayingframe, the identity of the current viewer can be used to select thefiltering limitations imposed. It is also within the scope of thisinvention that a default set of filtering limitations are imposed shouldthe viewer not be identified, multiple viewers be detected, or any otherprespecified set of conditions. Constraints imposed on the quality ofthe photograph to be displayed can be implemented, but also constraintsthat filter based on the sender’s (photograph capturer’s) desires;namely the sender imposed restrictions on the photographs to bedisplayed on the receiving frame. The receiver, namely the frame’sowner/user, also has the ability to limit the photographs to displaybased on their image quality or on their content.

The frame’s owner/user provides the sender access to the frame. Thisaccess, without any receiver side filtering, aside from image qualityconstraints, gives the sender control over the content to be sent andcorrespondingly displayed on the receiving frame. By allowing thereceiver to limit those photographs that are to be displayed, a greatersense of comfort and security are provided to the frame’s owner; hence,the frame’s owner is more likely to allow additional senders to accessthe frame, and thus, the community associated with the given frame islikely to grow. Receiver-side filtering rules can include, but are notlimited to: general content restrictions, such as but not limited torestricting the display of nude photographs or photographs where toomany people are included; individual specific restrictions, such as butnot limited to restricting any photographs that include a previousspouse; or restrictions relying on metadata constraints, such as, butnot limited to, restricting photographs taken prior to a certain date ortaken at a certain location.

FIG. 2 illustrates an embodiment of the invention includingreceiver-side photo filtering. In FIG. 2 , Susan is at home with herdigital picture frame 20. Susan’s daughter Sally and Sally’s husband Bobare in a remote location, where Bob takes a picture of Sally on Bob’smobile device 29. Bob is a community member with Susan (and both withSally), and Susan has set content constraints to allow pictures of Sallytaken by any community member to be displayed on her frame 20. Inembodiments of this invention, when Bob takes the picture of Sally, thepicture of Sally is automatically identified on device 29 and routedover a network, such as via social media server computer 26, to Susan’sframe 20 for display.

To route the picture to Susan, the content of the digital photo isautomatically identified, and any target electronic device, such asSusan’s frame 20, that would receive the photo also needs to beidentified and matched to the digital photo. The identified content isautomatically compared to content constraints of Susan’s frame 20 todetermine if the digital photo should be routed to Susan’s frame 20.Likewise other community member devices 29′ can be similarly processedeach time a photo is taken thereon to determine which, if any, photomatches or meets Susan’s content constraints and should be routed anddisplayed on Susan’s frame 20. Thus Susan can automatically receive, forexample, photos of each family member automatically upon the photosbeing taken. Susan can control the content and quality of the photos, toexclude, as examples, photos taken by community members of or includingnon-community members, poorly ‘angled’ selfies, photos taken with orthrough particular apps (e.g., Snapchat®), fuzzy, grainy or poorly litphotos, or duplicate photos (e.g., photos taken within a predeterminedtimeframe of each other). Photos not meeting Susan’s content constraintsare desirably not routed to Susan’s frame 20′.

In embodiments of this invention, content identification and/or arouting decision is/are determined by a content restriction filter. Thecontent restriction filter can be implemented via software coding storedand executed on, referring to FIG. 2 , the device 29, server 26, and/orframe 20. In embodiments of this invention, the filter components can beimplemented in modules on more than one device. For example, a photoquality filter module can be implemented on the photo-taking device 29,to limit photos sent to the server 26. The server can then performcontent identification on photos received, and route each photo to thetarget frame for which the photo meets the content constraints. Photoprocessing, such as cropping, aligning, scaling, enhancement, andrepetition filtering can be performed on the frame, server, and/ormobile device, as needed.

The digital photo frame 20 can additionally detect via the cameraSusan’s physical response, for example a body movement or facialexpression, to the routed/displayed photo, and an interest level gradebased on the physical response can be associated with the photo. Theinterest level grade can be used to determine whether the digital photois replayed, or a frequency at which it is replayed. The interest levelgrade can additionally be used to automatically update the contentconstraints, such as if Susan indicates a low interest to one or morephotos of Sally with her tongue out, or with Sally at a particularlocation (e.g., a bar). In embodiments of this invention, the frame candisplay a message or message indicator requesting the physical response,for example so the frame can learn what photos to display.

FIG. 2 further illustrates a second digital photo frame 20′ of Susan’sfriend Mary, remote from the frame 20. In an exemplary embodiment, Maryis a community member with Susan, Sally, and Bob, but Mary has notadded/allowed photos of Sally or Bob in the content constraints of frame20′. However, Mary loves Paris, and wishes to see all photos related toParis from her community members. If Bob’s picture of Mary with mobiledevice 29 occurs in front of the Eiffel Tower, then the contentrestriction filter module(s) will allow the photo to route to Susan’sframe 20 because Sally is present, and will also route the digital phototo Mary’s frame 20′ for including content related to Paris. In this way,community members can learn of activities of other community membersthat are of particular interest, without receiving every photo from theother community members.

A non-limiting embodiment of this invention consists of at least onepicture capture or storage electronic device 29 communicating directlyvia a network or wired or other wireless connection with the digitalpicture frame 20. To support such interaction, the picture capture orstorage device 29 connects via the network to the digital picture frame20 and downloads the digital picture frame interaction software to thedevice 29. Once installed, the application software scans all theavailable pictures on the device. The remaining process description isillustrated using a non-limiting example based on face selection. It islikewise, however, within the scope of this invention to select based onobjects other than faces including but not limited to scenes, materials,and activities. A second device 29′ (or more) can likewise link to theframe upon being in connection proximity.

Once selected, all photos containing faces are clustered by theapplication, either at the server or frame level. Individual photographscan then be automatically filtered based on features. Some of thesefeatures are quality related, e.g., red-eyes or blurred; some featuresare content related, e.g., excluding certain activities, people, orlocations. Those photographs that remain selected can be tagged withadditional metadata. Metadata includes all photograph generatedinformation, such as but not limited to location, time, or weather, aswell as derived data, such as but not limited to cluster identity.

The invention can use clustering to form grouped clusters according tophoto content. The clustering is desirably performed as a function of acommon detected content in the photos, such as things, places,activities, or combinations thereof. The clustering of photos can beused for determining a photo slideshow on the digital display of thephotos from a digital photo collection as a function of the clustering.The grouped cluster can be integrated to a slideshow as preferredimages, or used to provide requested slideshows, such as “show grandma”.

Embodiments of this invention include identity clustering, such as bylongitudinal facial clustering. Photographs are clustered not only inaccordance to an individual identity at a given time, namely at a givenage, but the disclosed longitudinal facial clustering clustersindividuals throughout their lifetime. Consider a collection of picturesthat span a prolonged period of time. In such a case, an individualnaturally ages, and creating a collage of that particular individualnecessitates accounting for their aging process. The disclosed approachuses photograph metadata, including but not limited to the date thephotograph was taken, to age the individuals to a common age using anyof the individual aging techniques known in the art. Once at a commonage, clustering is performed to group pictures of the same individual.

FIGS. 3 and 4 illustrate a clustering of photos of Queen Elizabeth. Inone cluster (“Young Cluster”) are photographs of the young Queen. Herage progression is represented in various sub-clusters forming a photochain to a second cluster (“Old Cluster”). Note however that thesephotographs are all of Queen Elizabeth, and as such, they are alleventually clustered together (as represented by the linking edges) toform a larger cluster. The sub-clusters can be linked together byfeatures that do not change as much over time, such as eye distance,head shape, nose shape, etc. Clusters according to this invention can beconstantly updated and adjusted, such as using new photos and/or othernew metadata.

An alternative or additional clustering technique uses statisticalmodels of metadata. Using metadata, such as, but not limited to,location identification or date and time, the likelihood of a givenperson being at a given location at a given time is computed. Shouldphotographs support such a given likelihood, they too are potentiallyclustered.

Similarly, metadata can be used to prevent the clustering of an unlikelyset of photographs. For a non-limiting example, image-processingtechniques are somewhat inaccurate at recognizing and disambiguatingchildren. This can be particularly true for infants and young children.Photo metadata that indicate when and where a photograph was taken canbe used to reduce or eliminate disambiguation. For example, photographsof children taken at a close time interval but at vastly separatedgeographical locations are likely to be different individuals.Similarly, children of a similar age but appearing in photographs takenfar apart in time are again likely to be different individuals.

Clustering in additional embodiments of this invention employs a logicaltwo-phase approach. It is within the scope of this invention that the“two phases” are integrated into a single phase, or are partitioned overa greater number of phases. Initially, a cluster is formed using facialrecognition using any one or more of the many known in the artclustering techniques. This phase clusters photographs of individualsthat closely resemble across all photographs. However, should anindividual modify their appearance either intentionally, say but notlimited to using a costume, or unintentionally, say but not limited to alopsided profile, an erroneous separate cluster may result. Independentof cause, the merging of clusters of the same individual is needed. Thesecond phase uses known image processing techniques to extract featuresof photographs within the given cluster. These features are used torepresent the cluster, say via a representative centroid. By averagingthe features across the photographs within the cluster, the effects ofminor variances in some of the features are diminished. The centroids ofall clusters are compared. If centroids of differing clusters aresufficiently close, the photographs within the close clusters arefurther examined. This further examination includes but is not limitedto the use of respective metadata. If still deemed similar, the clustersare merged.

In another embodiment, the disclosed photograph clustering approachcombines clustering with classification. Current clusters form the basisfor inducing classification models. These models, in turn, are used toassign currently unassigned photographs to existing clusters if thesimilarity measure score employed meets or exceeds the discriminativethreshold. It is within the scope of this invention to employ any of themany known in the art image similarity measures.

In another embodiment, the linking of similar photographs across framesor digital photo collections is performed. Frame and/or mobile devicephotograph collections are maintained separately, however the submissionof pictures to a frame can constitute a community membership. Withinsuch a community, photographs of individuals portrayed might be common.In some embodiments of this invention clusters of photographs acrossframes/collections are compared and the server suggests to the owners ofthe frames, as well as to those submitting pictures to the frames, tosend photographs of common individuals to frames that already possessphotographs of those individuals, but not necessarily the identifiedphotographs. Thus, photograph sharing is fostered. It is also within thescope of this invention to impose a threshold on the minimum number ofphotographs that a particular individual must appear in a remote frameor other digital photo collection prior to suggesting that additionalpictures of that individual be routed to that remote frame.

Another embodiment clusters photographs based on common detectedobjects. Exemplary objects include artifacts (things, places,activities) and living beings (typically people and pets). Locations andactivities can be detected via metadata and individuals present can bedetected via facial recognition.

Embodiments of this invention incorporate split displays for displayingtwo different photos simultaneously side-by-side. The photo displaysdesirably alternate between single and dual (or more) photo displays.FIG. 5 illustrates a digital picture frame 520 in a split screen mode,according to one embodiment of this invention. The digital picture frame520 includes a digital display 522, desirably sized with an aspect ratio(e.g., a 10.1 inch, 16:9 screen) beneficial to supporting both singlephotos and two side-by-side photos, mounted within an edge frame 524. Acamera 525 is connected to and integrated in the edge frame 524. Thepicture frame 520 includes any feature(s) described herein, such as anetwork connection module that connects over a wired and/or wirelessnetwork to mobile devices and/or a social media server computer forautomatically loading photos stored within one or more social mediaaccounts. The picture frame 520 further includes an automated displaymodule in combination with the network connection module and adapted toautomatically change photos displayed on the digital display 522 tophotos from an electronic device and/or a social media account of theowner, viewer, and/or one or more community members, such as uponautomatic detection with the camera of the viewer.

The invention includes methods of photo-pairing for a two or more photodisplay. The photo pairing can be random or intuitive, such as pairingsimilar photos. Similar photos can be identified based upon photocontent, including, but not limited to a person, more than one person,object(s), scenery/background/seasons, position of photographedobject/person, and/or place. Photo metadata can also be used formatching. The pairing can be based upon one matching person, such aseach photo including two different second persons (e.g., relatives) orin two different times/places, or two matching persons, such as in twodifferent times/places. For example, the photo pairing can include onematching person, with each photo including the person and one of twoother, related persons (e.g., mom and dad, different grandparents,different siblings, different pets, etc.) or places (two differentgraduations, different homes, different vacations, etc.). The inventionalso includes now and then side-by-side pairing, such as based upon agedetermination and clustering discussed above, where an older photo withone or more persons is paired with a newer photo of the one or morepersons. A slideshow that changes one or both sides of the dual displayis also available by this invention, such as showing a baby photo withan adjacent slideshow of the person at other ages (monthly, annually,etc.).

Methods of this invention include identifying two different photos inone or more digital photo collections that include a matching photocontent, and displaying the two different photos simultaneously in asplit screen mode on the digital display. FIG. 5 shows a display 522with a screen digitally divided into two equal sections. A first photoof the two different photos is in a first section 530, and a differentsecond photo is in a second section 532. As illustrated, the matchingphoto content between the two different photos is a same person. Thesame person is shown as an infant in the first section 530, and atgraduation in the second section 532. The two different photos are thusphotographically captured in two different calendar years, wherein thetwo different calendar years are separated by more than one additionalcalendar year, and paired or linked using methods described herein.

The matching of these photos can be done by the age-based clusteringdescribed above. Embodiments of the invention include automaticallyclustering digital photos of the digital photo collection according to acontent, and automatically selecting the two different photos as afunction of the clustering. The two different photos can be selectedfrom the same or two different photo clusters, such as on one or moreelectronic devices (e.g., both mom’s and dad’s separate clustered photocollections).

Embodiments of this invention include automatically clustering photos ofthe digital photo collection over a timeframe for display by preparing aplurality of sub-clusters, each for a common detected content in thephotos selected from things, places, activities, or combinationsthereof. Each of the plurality of sub-clusters is formed of the commondetected content at a similar time and/or age. Related sub-clusters ofthe common detected content, such as over a longer time period, arelinked together to form a cluster of related common detected content fordisplay, such as a function of one or more predetermined features havinglimited change over a timeframe. A first of the two different photos forthe dual display can be selected from a first of the relatedsub-clusters and a second of the two different photos from the second ofthe related sub-clusters.

FIG. 6 shows the frame 520 with additional photos displayed. In FIG. 6 ,the matching photo content is a same person and each of the twodifferent photos further includes one of two different second persons.The same person is a young girl, shown as an infant with her grandmotherin section 530, and at a second age (e.g., a teen) with her grandfatherin section 532. The photo pairing thus uses age-based cluster matching,and determines two different second persons that are related familymembers. The two photos can originate from a same or linked cluster(s),from a same device or interaction application, or from two differentclusters, such in two different devices or interaction applications. Asshown in FIG. 6 , the photo of section 530 is obtained from thegrandmother’s device 540, and the photo of section 532 is obtained fromthe grandfather’s device 542. As an alternative example, both photoscould be obtained from a photo collection of the mother of the girl.

Photo data can be preprocessed on the photograph-capturing device. Theclustering and cropping approaches described herein assume theavailability of the data on a server However, given current computingcapabilities on the photograph capturing devices, such as but notlimited to GPUs, significant preprocessing can immediately occur on theactual device. In doing so, the bandwidth required to transfer thephotographs to the server is vastly reduced. In embodiments of thisinvention, preprocessing includes, but is not limited to: image qualitydetection and enhancement (if photographs remain of poor quality thenexclude); content filtering (for example, pornographic or choppedfeature images are excluded); and repetitive (same picture takenmultiple times, only highest of quality is sent).

Additionally, picture fingerprints (compressed or very low resolutionversions) can be generated on the capturing device, and only thefingerprints are initially sent to the server. If the server deems thesepictures as desired, based on any pre-established criteria, the serverinforms the device, and the original pictures are transferred. Usingsuch an approach, only desired pictures are sent to the server, againreducing the bandwidth involved in the transfer.

Cross capturing-device filtering can also be incorporated. Using picturefingerprints, devices can interact directly among themselves todetermine which photographs to transfer to the server. For example, aset of community members may wish to strictly share images of commoninterest, and in embodiments of this invention the devices can sharefingerprints and automatically filter photographs of similar nature.Thus, only one of the photographs, potentially evaluated to be of thehighest of quality among the similar pictures, is transferred to theserver, again reducing the transfer bandwidth. Additionally, a votingprocedure can be imposed where only those picture with a sufficientlyhigh vote are transferred.

Likewise within the scope of this invention are other criteria by whichto select the photograph to transfer. Non-limiting additional oralternative criteria include but are not limited to the poweravailability or bandwidth capacity of the capturing device. As anexample, if a device has low bandwidth capacity or is nearly out ofpower, another device might be selected to send its version of a similarphotograph.

Photographs are posted to frames for the pleasure of the receiver,namely a viewer of the frame. In embodiments of this invention, theframe provides the frame viewer with the capability to provide feedbackto the sender of the photograph. Via a physical response, such as a handgesture, the viewer can send, if they so wish, an indication of thedegree of excitement or dissatisfaction with a given photograph. Oncethe gesture is captured and interpreted, the frame sends the server amessage indicating the viewer’s interest level in the photograph, whichin turn, is stored and applied at the server level and/or indicated tothe sender of the photograph. Such a feedback loop motivates the sendingdevice or collection to automatically send or not to send similaradditional photographs. Likewise using facial recognition, the frame canidentify the viewer, and if desired, tag the feedback with the viewer’sidentity. Also the frame can infer feedback expressed via facialexpressions using facial recognition. Similarly, other body movementgestures, such as but not limited to the nodding or shaking of headrepresenting positive or negative feedback, respectively, can becaptured and treated as feedback.

The invention further includes ranking the photos to be circulated onthe frame, such as for the purpose of automatically determining an orderof presentation, a number of display repetitions, and/or a time periodof display for each of the photos as a function of the ranking. For anygiven frame, the photographs sent to the frame are circulated. Theordering of presentation of photographs for a given frame can be basedon a ranking. In one ranking, similar photographs are clustered basedon, for example, but not limited to people, places, things, oractivities. Photographs within these clusters are then shown insuccession or within a predefined, or viewer set, separation of eachother. In another ranking application, feedback provided by the frameviewer(s) is used to determine the ordering, with bias given towards themore favorable photographs. In yet another ranking application,photographs are presented in a random order. In yet another rankingapplication, popular photographs, based on feedback, are repeated,possibly even frequently. In yet another ranking application, allphotographs are shown prior to any repetition. In yet another rankingapplication, the time stamp of the photograph is alternatively oradditionally used for ranking. That is, a chronologically ascending ordescending order of photographs can be presented. In yet another rankingapplication, photographs are displayed according to similar seasons ordates within successive or separated years. In yet another rankingapplication, a popular photograph is displayed longer prior to beingreplaced by its successor. In yet another ranking application, theduration of display is the same for all photographs. In yet anotherranking application, photographs are clustered according to time slices,the number of photographs within each time slice is computed, andphotographs within popular time slices are shown more frequently. It iswithin the scope of this invention to vary the time slicessignificantly, whereby brief time slices with high photograph countstend to indicate great interest in the shown event, location, orindividuals. Lengthy time slices, without loss of generality, canaccount for trips, seasons, or any sustained activity or event. In yetanother ranking application, the viewer explicitly specifies theordering and duration of display. One skilled in the art recognizes thatany ranking rule known in the art can be imposed and that thesepresented rankings serve only as non-limiting examples. More so,multiple combinations of these and other ranking schemes are likewisesupported by this invention.

Another embodiment of this invention includes, and the digital pictureframe is implemented with, a method, system, and/or apparatus, such asembodied in an MPSM or other software application, that automaticallydetermines and shares a location, an activity, and/or photos of a user.The application learns user activity over time, with the learning basedupon user locations and/or context. The application can learn throughautomatically determining activities at locations based upon knowncontext information and past context information for the location. Theapplication can tag photos for determining context relevancy for showingon the digital picture frame, as discussed above. The invention furtherincludes energy saving location methods for the mobile device that canbe used to more efficiently allow the location and social media aspectsof the invention to be implemented on a mobile device. The method andapplication can be used for any suitable function, such as a safetyand/or reminder serves, and is particularly useful for use in socialmedia applications and for generating photos for display on the digitalpicture frame. The invention will be described below with implementationin an MPSM system, and particularly with an MPSM application that learnsuser activity over time, with the learning based upon user locationsand/or context.

The MPSM method and system of this invention is mobile and positional innature. Such systems, like many other systems originally developed onone type of computing platform but migrated to another, operate not onlyon mobile environments. That is, while MPSM implementations are targetedto primarily execute on mobile devices, such as but not limited tosmart-phones, tablets, and/or laptops, they often support implementationfor non-mobile environments such as but not limited to desktops andworkstations, servers, and large scale compute farms and cloud computingservers. The invention will be described below with a mobile device,such as smart phone having cell service, a GPS system, and access to theInternet via WiFi.

The MPSM method and system of this invention is desirably executed orimplemented on and/or through a mobile device computing platform. Suchcomputing platforms generally include a processor, a recordable medium,an input/output (I/O) device, and a network interface capable ofconnecting either directly or indirectly to the Internet. The mobiledevice executes over a networked environment, a non-limiting exampleshown in FIG. 7 . The mobile device is connected, either directly orindirectly, using any of the many techniques and technologies known inthe art, over a network, to back-end system or systems,itself/themselves computing devices. The mobile device can connect witha remote server, shown in FIG. 7 as server 38, to store and/or accessuser or community information.

MPSM systems are used to support users remaining socially aware of theircommunity. That is, their primary usage typically is to actively monitorthe location and activity of family members, friends, colleagues, andgenerally others within one’s community. Communities can be partitionedinto sub-communities where the union of the sub-communities forms theuser’s community. The sub-communities may or may not overlap. Thepartitioning of communities into sub-communities is beneficial insupporting specialized applications. For example, while a user mighthave general interest in the location and activity of all of theircommunity members, they might be particularly interested in the locationand activity of those who might be suddenly in need of assistance.

The creation of a community can include the issuing of invitations. Aninvitation is a request by a user A of another user B to allow theinviting user, user A, to track the activities of the invited user, userB, and vice versa. If the invited user accepts, the inviting and invitedusers form a community.

A community is relevant to only that user which formed it. That is,different users have different communities. A community is a grouping ofinvited (referred to as remote) users by the inviting (referred to aslocal) user. A local user can partition or merge a community, thusforming a sub-community or a parent community, respectively. Forexample, consider 5 users: Bob, Sam, Sally, Alice, and Susan. Bob caninvite Sam, Sally, and Alice, thus forming his user community. Bob canlikewise partition his community into a sub-community consisting of onlySam and Sally. Sally can invite Susan. Thus, Sally’s community wouldinclude Bob (via his invitation) as well as Susan. If no additionalinvites occurred, Sam’s and Alice’s respective communities would onlyinclude Bob (each via Bob’s invitation), while Susan’s community wouldonly include Sally (via Sally’s invitation).

Providing users with the opportunity to expand their communities in aconvenient manner is advantageous. Such expansion can seamlessly beaccommodated by including users listed in a user’s contact lists eitheras a whole or selectively into their community. Contact lists include,but are not limited to, users listed in a user’s local address book,e-mail contact list, Twitter follow list, LinkedIn connections list,and/or Facebook friends list. By incorporating users listed in a user’scontact list, the user’s community is expanded without effort. Note,however, that selected inclusion can be supported; thus enablingcommunity growth without unnecessarily over-expanding the community.That is, entries from the contact list can be included in their entiretyand the user can selectively remove those entries which s/he wishes tobe excluded from the community. Similarly, entries from the contact listcan be selectively added.

Users are identified by their account identifier. To use MPSM a useraccount is created. User accounts generally require a user login, whichis a unique user identifier, and a password or equivalent. After havingcreated an account, a user can log in. Initially, the local user doesnot have a community. In embodiments of this invention, over time, themethod and application tracks the activities and location of the localuser. Should the local user establish a community as aforementioneddescribed, the community members will likewise be tracked. Local usersreceive notifications of the location and activities of their communitymembers. Once logged in, the local user can select to activate ordeactivate self and community tracking and notification. If notoverwritten, default settings are used.

Whenever logged in and tracking is enabled, a user’s location andactivity is tracked. That is, a user periodically records their locationand/or activity. Locations are tagged by name. Names can be but are notlimited to the following schemes: physical (e.g., 123 Oak St.), absolute(e.g., Acme Coffee), and/or relative (e.g., my work office), orproximity (e.g., two miles from home). Activities are typically events.These events might be common to the entire community such as: “drinkingcoffee,” “eating lunch,” “sampling wine,” “working from home,”“commuting,” etc., to more specific to a local user such as “restoringcar” or “driving to lake home.” Multiple activities can occursimultaneously. Users can change their activities at any time. Unlesspreloaded or derived from an external source, such as but not limited toa location database, initially, all locations and activities areunknown. Local users must record all such location-activitycombinations, i.e., a local user must name or tag the location and theassociated activity. A list of activities common to the local user’scommunity can be provided. This community activity list can be rankedeither arbitrarily (randomly), according to most recently used, mostfrequently used, relevance to location, alphabetically, etc. Eventually,an activity list specific to the local user is learned. This local useractivity list can be displayed to the local user either individually,along with the community list, or merged with the community list. Again,any of these lists can be ranked as previously mentioned.

FIG. 7 illustrates a representative area 30 to demonstrate a method ofand application for locations and/or activities of a user participatingin a social networking service. The area 30 is shown as a cellularcommunication network including a plurality of cells 32 each disposedaround a cellular communication antennae or base station 36. Within thearea are a plurality of destinations each shown as including a WiFiInternet connection. The local user has one or more electronic devices,such as a mobile device that communicates with a remote server 38 viathe cellular network and/or the WiFi connections. As will be appreciatedthe methods and applications of this invention can operate within anysuitable size and configuration of the communication area, depending onwhat the user encounters.

Destination 40 is the home of the user. The user commutes to office 42for work on most business days. On the way the user typically stops atthe coffee shop 41. For lunch on most days, the user visits restaurant43, but on Wednesdays the user typically meets a second user for lunchat restaurant 44.

At each destination 40-44, the user enters user information about thedestination. The application and computer system that receives the userinformation automatically associates the user information with thedestination, and stores the user information in a locations database,such as on the device and/or at server 38. The destination desirably isdetermined automatically and tagged with the user information, such as alocation name of the destination and/or the user activity beingperformed at the destination. For example, destination 40 can be taggedas “home” and likely has numerous activities associated with it. Thedestination 41, and any photos taken, can be tagged as its establishmentname “Acme Coffee” or simply “coffee shop” and associated with the useractivity of “buying coffee” or “latte time.” The manually entered userinformation can then be automatically shared to the user’s community ina social networking service. Similar user information is received forthe other destinations 42-44. The user information desirably includesany other information about the location or activity, whether manuallyentered or automatically determined, such as the time of the visit oractivity. Some destinations, such as home or work will likely havemultiple user activities over a period of time, such as “coffee break,”“meeting time,” and/or “quitting time.”

The computer system receives user information and associates the userinformation with the corresponding destination, and any photos taken,for multiple visits to each of the destinations 40-44. The computersystem begins learning the locations and user activities. In embodimentsof this invention, the user can be automatically prompted forconfirmation of the user information upon arriving at a destination toconfirm the location and/or user activity. For example, the user can beprovided with an automatically generated list of previously entered useractivities for the destination upon arrival, thereby promoting efficientcollection of information. The items on the list can be listed in anorder based upon a particular ranking, such as number of times enteredpreviously or based upon a context, such as what activity is likelybeing performed at a particular time of a particular day.

Over time, the computer system learns the user information and beginsautomatically associating and identifying at least some user activitiesfor corresponding locations and any photos taken. As will beappreciated, the automatic identifying of activities at locations willlikely occur at different rates for different activities and locations,with some locations having fewer activities and/or more frequent visitsthan others. In preferred embodiments of this invention, the systemautomatically shares the user information in a social networking serviceupon automatically detecting further user arrivals at the destination.Photos taken are likewise automatically tagged with the userinformation. The automatic sharing of user locations and/or activitiesdesirably occurs upon the user’s arrival at the location, or at aparticular time at the location. As such the invention includes anautomatic detection of the user’s arrival at a destination. Theautomatic sharing and photo tagging desirably operates without useraction and prior to receiving any additional user information for thedestination.

As an example, the user may typically purchase lunch at destination 43,but on Wednesdays typically goes to lunch with a friend or spouse atdestination 44. The lunch routines of the user, and particularly theWednesday lunch routine, can be learned by the system and automaticallyshared to the user’s community upon the system automatically determiningarrival, without manually input from the user. If the user is havinglunch with a community member, then the system can automaticallydetermine that both users are at the same location together toautomatically recognize and confirm the lunch activity, and proceed toautomatically share the information for both user’s to their respectivecommunities. If the user deviates from a routine, the system can knowthis, and refrain from sharing the typical destination, by the mobiledevice detecting a different location than the typical routinedestination.

In embodiments of this invention, learning is accomplished by any knownmachine learning, data mining, and/or statistical techniques known inthe art. Supervised, semi-supervised, and/or unsupervised approaches canbe used, including, but not limited to Naive Bayes, Neural Networks,Support Vector Machine, and/or Associating Mining based techniques.

The MPSM method and system of this invention desirably records allposted locations and activities. Throughout use, the disclosed inventionlearns the corresponding locations and the set of associated activities.More so, via comments made by the local user and by the local user’scommunities, the importance of the activities can be learned, such asfor the prompting discussed above. Importance can be either local useror community biased. Additionally, importance can be biased by context.For example, community members as a whole might prefer “eating steak,”“eating pizza,” and “eating sushi,” in that respective order. On theother hand, a local user might only eat sushi. Thus, local user biaswill yield “eating sushi” only, while community bias will suggest“eating steak,” “eating pizza,” and “eating sushi,” in that respectiveorder.

In embodiments of the MPSM method and system of this invention,locations are named according to a naming convention. Regardless of thenaming convention used, a location is a physical geographical position.More so, physical geographic locations associate properties that canvary with or be dependent on context, namely time and date (hours, dayof week, calendar date, etc.), users involved, and their relationshipsto each other, etc. This context can affect the associated location nameor activity.

A common scheme that can be used to at least assist in identifying aphysical geographical location is via the use of geocoding. Geocoding isthe representation of a physical location via the pairing of latitudinaland longitudinal coordinates commonly referred to as a lat-long pair.Global Positioning Systems (GPS) can also determine a physical positioncoordinated via the triangulation of satellite transmissions. TypicallyGPS devices derive lat-long pairs which are made available to a varietyof applications, often via map displays. GPS economics, accuracy, andsimplicity of use resulted in their wide appeal and commercial success.Their continuous use in mobile devices is problematic, however, as theyare energy intensive and rapidly drain the battery. Thus, alternativemeans or approaches to detect locations are desired.

Embodiments of the MPSM method and system of this invention, asdiscussed above in FIG. 7 , use or rely upon cell coordinates. Whenmobile devices communicate with a cell tower, they send their cellcoordinates. These coordinates are recorded by the cell provider and aretypically not publicly known. The cell phone or, in this case, themobile device supporting the positional social media system, however, isaware of their coordinates. Thus, the device can store the cellcoordinate position and automatically associate that cell coordinatewith the location name provided by the local user. Over time, a locationdatabase of cell coordinate and named location pairs is created. Thelocal portion of the database favors the local user. The union of allthe local portions of the location database desirably constitutes thename space of the entire MPSM system of this invention. It is understoodthat any of the many database management systems or storage schemesknown in the art can serve as the platform for this location database.Thus, location names can be provided without the need to rely on aglobal positioning system, reducing battery consumption. Location datacan additionally or alternatively be purchased or otherwise provided bya third party.

An additional and/or alternative approach for automatic locationdetermination relies on WiFi triangulations. Mobile devices can grow andmaintain a database of known open WiFi networks, for clarity we callthis database an Open-WiFi-Net database. Such mobile devices can use theinformation stored or derived from the information stored in theOpen-WFi-Net database to further refine the accuracy of a locationwithout the use of GPS. Via point triangulation, when an Open-WiFi-Netdatabase is available, the mobile operating system uses not only thecell tower but also WiFi triangulations to determine location. It iswithin the scope of this invention to use either or both cell-phone andWiFi triangulations to enhance location information in addition to anyother disclosed approach. The mobile device can use the WiFi signal at adestination, such as destination 43, and additionally or alternativelyany detectable open WiFi signal from a neighboring location, such asestablishment 45 that is adjacent destination 43.

Having created the location database, searching, namely querying, thedatabase uses the cell coordinate or the location name. That is, alocation name query takes a location name as input and returns thecorresponding cell coordinate. A cell coordinate query takes a locationname as input and returns the corresponding location name. Note that,multiple names can be attributed to a given cell coordinate. That is, alocal user might name a location using multiple different names;different users can name same locations using different names.Similarly, the same name can be used for different cell coordinatelocations. All names corresponding to a given cell coordinate arereturned. It is within the scope of this invention to selectively returnnames based on context, user, or community bias. Similarly, all cellcoordinates corresponding to a given name are returned. Again, it iswithin the scope of this invention to selectively return coordinatesbased on context, user, or community bias. Ranking of the resultsreturned can, when desired, be biased towards the local user.

A key concern for MPSM systems is collecting location information.Clearly any location information available within the mobile deviceshould be harnessed. Thus, if GPS readings or any other locationinformation is generated by other device resident applications, thesereadings are desirably recorded and utilized by the method andapplication of this invention. However, reliance on strictly otherapplications to obtain positional information is obviously not realisticor possible.

In embodiments of the MPSM method and system of this invention,positional information is obtained via the use of geofences. A geofenceis geographical boundary or “fence” surrounding a positional reading. Asthese boundaries are radius based, geofences are generally circular.Location transmission occurs whenever a handover of one cell tower toanother occurs and is expected but not guaranteed to occur once ageofence boundary is crossed. To track location, periodic locationtransmissions are required. Since location transmissions must beminimized to conserve device energy, transmissions should only occurgiven geographical movement. Thus, crossing a geofence should generatesuch a transmission. Unfortunately, as crossing a geofence does notguarantee a location transmission, increasing the likelihood of atransmission is necessary.

In contrast to the known uses that surround a location with a singlegeofence, to increase the likelihood of a location transmission duringmovement, embodiments of this invention include surrounding a locationgeofence with a plurality of geofences. In one embodiment of thisinvention, a method and system of tracking a user includes determining alocation of the mobile user, automatically establishing a first geofencearound the location, and automatically establishing a plurality ofadditional geofences around the first geofence, with each geofenceincluding a boundary. A location transmission is obtained by the mobiledevice upon crossing a boundary of the first geofence or any of theplurality of additional geofences. Multiple neighboring geofences areadvantageous since they increase the likelihood of a locationtransmission as their boundaries are likewise likely to be crossed givenmovement.

FIG. 8 representatively illustrates a geofence 60 surrounding a currentlocation 62. The geofence 60 is surrounded by additional geofences 64,all within a given cellular tower transmissions cell 65. Note that partof a neighboring geofence 64′ is not fully within the cell 65, andhence, limits its benefits since a cell tower handoff by movement intocell 65′ will generate a location transmission.

Geofences are implemented as software processes. Operating systems formobile devices, such as but not limited to iOS and Android, limit thenumber of processes available to an application, and thus, the number ofgeofences is bounded. However, this limit typically exceeds the numberof geofences generated using the approach described above. Therefore,additional processes are available, and hence, additional geofences arepossible.

To increase the likelihood of a location transmission given movement, inembodiments of the invention, the remaining available processesimplement static geofences. A static geofence is not dynamicallygenerated given a new location as previously described. Rather, a staticgeofence is one that is fixed and represents those locations that arelikely to be crossed by a given user. That is, users are habitual andfrequent a limited set of locations often, for example but not limitedto, their home, office, or favorite wine or sushi bar. By learning thefrequent locations of users both individually and system wide andsetting static geofences at these locations, biasing by the individualuser, the probability of a location transmission is increased sinceadditional geofences are likely crossed.

More so, these repeated locations vary by city, county, state, country,etc., as well as by other factors such as but not limited to day andtime. Geographical and temporal presence can thus be used to vary theset of static geofences for a given user. For example, the set of staticgeofences for a given user will vary if the user is in Washington, DCrather than in San Francisco, CA. Similarly, the set of static geofencesfor a given user will vary depending on the day and time. For example, auser frequents work on weekday mornings but frequents their favoritebagel shop on Sunday mornings and their favorite sushi bar on Thursdayevenings.

Location transmissions suffer from a margin of error. Thus, it isdifficult to precisely pinpoint and tag a location with a singletransmission. Embodiments of this invention include automatic refiningof a location of a user destination as a function of user routines, suchas established by several user visits to the destination. As timeprogresses however, and a user frequents the same location multipletimes, multiple location transmissions for the same location arerecorded. In one embodiment of this invention, as representatively shownin FIG. 9 , by overlapping the transmitted location along with itsmargin of error, a more accurate location can be derived. Theoverlapping of location transmissions for a given location 70 betweenstreets 72 and within geofence 74, along with their margin of errors,represented as circles 76, yields an accurate location placement.

As shown in FIG. 9 , location accuracy improves as related data arecollected. Related data, however, can, at times, be somewhat erroneous(in terms of accuracy). A non-limiting example is an entrance to ashopping mall. Such an entrance is not necessarily at the center of thecomplex. Regardless of the entrance displacement from the center of thecomplex, the entrance location can still be used to increase locationaccuracy of the mall complex since the readings for the entrance areconsistent. That is, for a given user, given mobile device, givencarrier, etc., such location recordings remain consistent, all be it,slightly erroneous. Thus, even dirty, namely potentially inaccurate,data can result in correct location identification.

Additionally, having established a location, corresponding lat-long paircoordinates can be reversed engineered, namely mapped back onto, a placename. These derived lat-long pair coordinates become yet an additionalinformation component that is used by a learning system to better refinea mapping to a named place. Machine learning, data mining, andstatistical approaches that are supervised, semi-supervised, orunsupervised can be used, as known in the art, to cross-correlate allavailable location related data.

Once determined, the user information including the location and/or theuser activities are automatically stored in a database. Embodiments ofthe MPSM method and system of this invention include a computer serverfor providing and implementing the tracking and/or social networkingservice of this invention. The computer server includes a locationmodule to determine the user location and/or a tagging module configuredto correlate manually entered user information to a user destination anda database module configured to store user information including userlocations and user activities at the user locations. For social mediaand photo sharing, the server further desirably includes a communicationmodule configured to automatically share a user activity or photo in thesocial networking service upon further user arrivals at a correspondingone of the user or community locations. The server can also include anassociation module configured to associate the user activity with thecorresponding user location and any photo taken.

Since location transmissions are needed during movement, the obviousquestion arises: when should the transmissions cease? That is, thesystem must determine when the user has arrived at a location to knowwhen to perform the automatic steps discussed above. As discussed above,GPS systems are an energy drain on a mobile device, particularly as theGPS remains on and linked with the satellites to maintain locationdetection. Keeping a GPS application operating is a drain on both theprocessor and the battery of the mobile device. This invention providesa method and executable application that conserves energy by notcontinually running during use of the mobile device.

Embodiments of the MPSM method of this invention provide an automatedmethod of tracking a mobile user that includes providing a locationmodule configured to receive location transmissions, placing thelocation module into a sleep mode, awakening the location module uponreceipt of a location transmission, and determining a location with thelocation module. These placing, awakening, and determining steps arerepeated, thereby placing the application into a sleep mode when notneeded, thereby reducing the drain on the mobile device. The applicationgoes into sleep mode when necessary or when desired, such as when theapplication is not needed, e.g., during extended movement or upon anarrival at a location. In embodiments of the MPSM method and system ofthis invention, the application can go into sleep mode whenever a timesince the device awakening exceeds a predetermined time allocation, orupon a determined rate of travel exceeding a predetermined threshold,thereby indicating extended travel.

FIG. 10 illustrates one exemplary, and non-limiting, method according toan embodiment of this invention to automatically detect arrival at adestination. The method is useful for tracking a user’s location for anyof various reasons, including, for example, for safety, to provideautomated reminders, and/or to provide automated suggestions to the userbased upon the destination and/or surrounding area. The method of FIG.10 is particularly useful for implementing the method and systemdiscussed above, and can be used to implement other applications andmethod to provide energy savings compared to GPS location methods inmobile devices.

FIG. 10 includes a flow chart 100 that includes and/or represents threedistinct situations, namely, an actual arrival, rapid movement, andsporadic movement without an actual arrival. Initially, the applicationis in sleep mode. Sleep mode is a state when no processing, and hence noenergy consumption, takes place. Processing occurs once the applicationis awoken. A location transmission, such as a cell tower transmission oranother application obtaining location information, awakens theapplication in step 102. Since the application awakening occurs due to alocation transmission, the current location is known.

Once awakened, the application typically has a maximum amount of time tocomplete its processing. This limit, called time allotment, is set bythe device operating system. All processing must complete prior toexceeding the time allotment. Ideally, the application should relinquishthe processing flag back to the device operating system before theoperating system forcefully removes the application from its activequeue. Voluntarily terminating an application, namely returning it tothe sleep mode, rather than having it forcefully terminated by the hostoperating system, is consider good citizenship. In step 104, theapplication initializes two timers, namely, a timer count representingthe duration of time the process has executed since last awakening, anda stationary count representing the duration of time since the lastdetected device movement.

As time progresses and the process executes, the timer count isincremented in step 106. In one embodiment of this invention, wheneverthe application processing time exceeds the operating system timeallocation (108 - YES branch), the application is voluntarily placed insleep mode 105. Note that the time allocation threshold is notnecessary, but set to support good citizenship.

Assuming that the time limit has not been reached (108 - NO branch), theapplication waits for t time units in step 110. After waiting t timeunits, new current location data are obtained is step 112 and storedlocally on the device in step 114. In step 116, the current location iscompared to the previously known location. If the two locations differ(116 - NO branch), the rate of travel is computed in 118. If the rate oftravel exceeds a threshold (120 - YES branch), the process is desirablyand voluntarily placed in sleep mode 122. Rapid travel is unlikely toresult in an immediate or near term arrival; thus, checking locationswhile moving rapidly unnecessarily uses device energy. Eventually, theapplication process is awoken with the device moving at a slower rate.At that time, location checking is needed as an arrival might soonoccur. If or when the rate of travel is slow (120 - NO branch), movementis noted in step 124, and the loop is repeated commencing with theindication that additional processing time has elapsed in step 106.

Thus far, the arrival detection process has been voluntarily placed insleep mode either due to having exceeded the self-imposed processingallotment quota which is desirably set just slightly below the operatingsystem’s time limit that leads to the removal of the application fromthe active queue (108 — YES branch) or having travelled too rapidly (120— YES branch). Slow travel has resulted in simply recording thelocations traveled, noting the movement exists in step 124, and awaitingeither arrival or process termination.

Arrival is determined when the same location is detected for asufficient duration of time. That is, an arrival is potentiallydetermined when the location remains the same (116 -YES branch). Thestationary detection count is then incremented in step 126. If thestationary threshold is not yet exceeded (128 - NO branch), theapplication waits for t time units in step 110, and the current locationis obtained in step 112 and stored locally in step 114. A sufficient andpredetermined duration at the same location eventually surpass thearrival detection threshold (128 - YES Branch).

Once arrival is determined, arrival is declared in step 130, all dataregarding the prior locations visited and stored locally are compressedand sent to the back end system supporting the application in step 132.A new location checkpoint is established in step 134, and the process isplaced in sleep mode 136. From the sleep modes, the process of FIG. 10repeats upon a known location.

Compression of location data is typically performed prior to localdevice to back-end system transmission as often the location data manynot be needed at the back end. Location data may not be needed in cases,for example but not limited, during rapid travel. Although exemplifiedas having data compression occur prior to the sending of the data to theback-end, it is within the scope of this invention to compress locationdata prior to storing them locally.

All parameters described above for FIG. 10 , for example t (for the timeunits), timer count, etc., are system and device dependent.Experimentation with and fine tuning of these and other parameters iswithin the scope of this invention. Also within the scope of thisinvention is the tuning of these and other parameters via the use ofmachine learning, data mining, and statistical approaches; supervised,semi-supervised, and unsupervised approaches can be used.

As discussed above, once the user has arrived at a destination, thelocation identification, user activities at the location, and/or anyproximate third party members of the user’s community are determined, ifnot already known. In this way, the devices automatically continuallydetermine locations which can be used to identify any establishmentsand/or any community members at or within proximity to the location.

User activities are actions or events. Example user activities includebut are not limited to “drinking wine,” “flying,” “reading,” “attendingconference,” or “commuting.” Users specify a particular user activityeither by selecting from a provided list or by entering a different useractivity. As discussed above, the provided list is generated by storingall previously entered user activities from all systems users butbiasing the ranking of the provided activities based on context, thelocal user, their community, or a combination thereof.

All location and user activity pairs are stored in a databasecorrelating the location with the activity. Any of the many databasemanagement systems or storage schemes known in the art can serve as theplatform for this location-activity database. Furthermore, it is wellunderstood in the art that the location-activity database can store manyadditional features. For example, the user identity and date and time ofthe pair are recorded.

Over time, the database grows and contains a sufficient number of pairsto support mining. The volume of data needed to mine correlations isdependent on the mining algorithm deployed and the level of accuracyneeded. As known in the art, there are many machine learning, datamining, and statistical approaches to support mining. By using any ofthe many available such approaches, either individually or incombination, a local user activity preference per location is learned.Example learning approaches include supervised, semi-supervised, andunsupervised approaches including but not limited to Naive Bayes, NeuralNetworks, Support Vector Machine, and Associating Mining basedtechniques. The use of proprietary mining techniques is likewise withinthe scope of this invention. Once local user preference is learned, thispreference is used to bias the aforementioned provided user activitylist.

There are many approaches to identify locations. Automated locationidentification is accomplished by periodic checking of the currentlocation. Periodicity of checking depends on, for example, the methodused to determine the location, the desired frequency of reporting,recording, and notification, and the resources available to support thechecking. Other periodicity determination approaches known in the artcan likewise be used. One approach to automate location identificationis the periodic determination of lat-long pairs via the use of a GPSdevice. An online service or a locally resident database can be used tocorrelate the GPS readings with locations. A preferred embodiment ofthis invention uses the aforementioned location database. Whenever atransmission to a connected cell tower is made, the cell coordinates ofthe transmitting device are used as a search query against the locationdatabase. If a match is detected, that location is identified. Anotherpreferred embodiment detects locations upon the crossing of geofenceboundaries as previously discussed. Note that both dynamicallydetermined geofence boundaries and static geofence boundaries detect alocation. Yet another preferred embodiment detects locations bycapitalizing on location transmissions generated by any otherapplication operating on the mobile device requesting locationinformation.

In embodiments of the MPSM method and system of this invention, localusers, unless disabled by a local user, can be provided with automatednotifications for themselves and for their community members. Thesenotifications describe locations, activities, or correlated locationsand activities for themselves and their community members. For example,unless disabled by the user, any time a user arrives at a new location,the local user and their communities can be notified of the user’s newlocation. Automated location detection and notification, unlessdisabled, occurs without requiring a local user prompt.

Similarly, activity notification can be automated. Once a user arrivesat a location, a set of activities previously occurring at that locationis shared with the community or provided to the local user forinformation or sharing. If the user chooses to confirm at least one ofthese past activities, both the local user and their respectivecommunity members are notified of this at least one activity, and anyphoto taken is automatically tagged with the context information.

In another embodiment of this invention, automated notification involvesshared experiences. A shared experience is one that associates multipleusers. These associations can be passive or active. A passiveassociation is strictly informative in nature while an activeassociation requests an action. Non-limiting examples of passive sharedexperiences based on locations include: “User A is at User A’s office,as is User B” and “User A is at home as is User C.” Note that the firstexample involves multiple users at the same physical location, namelyUser A’s office, while the second example involves multiple users at thesame relative locations, namely their homes, but at different physicallocations.

Similarly, passive shared experience notifications can be based on useractivity. Non-limiting examples of passive shared experiences based onactivity include: “User A is eating lunch as is User B” and “User A isparticipating in her favorite sport as is User B.” Note that the firstexample involves multiple users participating in the same activity,namely eating lunch, while the second example involves multiple usersinvolved in similar nature of activities, namely participating in theirown favorite sport, which can be different actual activities, namelyracquetball and swimming. In both passive shared experiences based onlocation and on activity, known in the art machine learning, datamining, and statistical approaches that are supervised, semi-supervised,or unsupervised approaches can be used to correlate relative locationsand activities to physical locations and activities.

Other shared experiences can prompt for action, and are thus consideredactive. A non-limiting example of an active shared experience promptingfor action includes: “User A posted a picture when at Penn Station; youare now at Penn Station; please post a better picture?” Thus, activeshared experiences request the user to actively react. As above, activeshared experiences can be location or activity based and can be absoluteor relative. Note that it is likewise within the scope of this inventionthat individual user notifications be active and passive, in a similarmanner as described above. However, the correlation of locations andactivities both for passive and active are based strictly on thecurrent, past, or projected expected activities of the individual userrather than those of multiple users.

Typically, only changed locations and activities are notified. That is,a location or activity is not typically repeatedly identified. However,a local user can request repetitive notifications based on anytriggering condition known in the art.

Local users do not always remember to indicate a new location name orconfirm which of the possible suggested name or names the systemindicated for the given the location. As such, it is at timesadvantageous to prompt the local user for information. However, overlyaggressive prompting might annoy the user. In embodiments of thisinvention, the application non-invasively prompts the user upondetecting an unknown location for the given local user. To avoidannoyance, prompting is repeated only rarely, say twice; the number ofrepeated prompts can be set as a parameter. Similarly, to provide asense of comfort, if the back-end system recognizes the location basedon the local user’s community members’ naming schemes, it prompts thelocal user with guiding messages, for example but not limited to “Manyof your community members call this location The Tasting Room”.

Identification of activities associated with a given location or a givencommunity member can be additionally or alternatively automaticallyinferred in multiple ways. In embodiments of this invention, thecomputer system can automatically determine a positional destination ofa user, such as by using a mobile device discussed herein, andautomatically deduce as user information a location type and/or useractivity of the positional destination. The user information can bededuced, at least in part, based upon the destination context. Exemplarycontext information includes, without limitation, time-dependentinformation (e.g., what time of day is it?), community information(e.g., who is also there?), and/or third-party information about thepositional destination. This method, tied with automatic sharing of theuser information in a social networking service, can provide a partiallyor fully automated process for determining user location and activity,and tagging photos taken with the context information.

In one embodiment of the MPSM method and system of this invention, theautomatic deducing of the user information is based upon known orlearned user routine. As discussed above, local users typically followstandard routines. Some routines are daily, weekly, monthly, etc. Otherroutines are context dependent. Regardless of the nature of the routine,learning via any of the many statistical, machine learning, data mining,or business analytical techniques known in the art, enables predictivebehavior and automated activity and location suggestion. For example,but not limited to, if a local user always goes out to lunch at noon onevery weekday, then if an unknown location is detected on a Tuesday atnoon, then the application can suggest that this unknown location islikely a restaurant and the activity is likely eating lunch. Similarly,routine identification enables the prevention of transmissions bothpositional and informational. For example, but not limited to, if alocal user always goes to sleep at midnight on Sunday through Thursdayand awakens at 7:00 am the following day, then energy can be saved ifthe application voluntarily places itself in sleep mode during the hoursthat the local user is known to be sleeping. Additionally, routines caninvolve a sequence of activities and locations. A non-limiting exampleof a sequence of activities includes: On weekdays, Eric arrives at hisoffice at 8:00 am, drinks coffee at 10:00, develops software from 11:00am until 5:00 pm, commutes home at 5:30, and finally, arrives at home at6:00 pm.

Another location and/or activity deduction approach is by association.The automated deducing can include automatically associating a user witha second user at a positional destination. If the second user’s locationand/or second user’s activity is known, then the system canautomatically infer the location type and/or user activity of the firstuser from the second user location and/or activity. Consider a previousknown event such as: “Community member Sally swimming at the Somersetpool”, assuming that the Somerset pool location was previouslyidentified. As an example of automatically determining a currentactivity of community user Sam, the system identifies through locationdetermination that Sam is currently at the same location as Sally, andalso that Sally is currently at the Somerset pool. From thisinformation, possible automatically postulated associations andactivities are: “Sam is at the Somerset pool”, “Sally is swimming”, and“Sam is swimming”. Thus, it is possible to infer an activity for acommunity member from association with another community member. It iswithin the scope of this invention to use any logical inference methodsknown in the art to generate plausible associations. It is also withinthe scope of this invention to obtain confirmation of the plausiblepostulated activity by the community member, in this case Sam, by askingeither Sam or Sally or by any other means known in the art.

Desirably the computer system operating the MPSM automatically storespast user information, including past location type and/or user activityof the positional destinations of all users. User information for futurevisits to repeat positional destinations can be automatically deduced asa function of the stored past location type and/or user activity of thepositional destination. In embodiments of this invention, the system canrely on recorded previous activities of a user, a community member, orany system user at a given location to postulate on a user’s activity ata given location. Past context information for past visits to thepositional destination by the user and/or community members of the usercan be compared to a current context of the user’s visit to thepositional destination to deduce the user information. In oneembodiment, the system can reduce possible location types and/or useractivities as a function of the past location type and/or user activityof the positional destination.

As an example, at a given Location A, users previously studied, talked,ate, and drank. Thus, if a user’s positional destination is detected asat Location A, then plausible activities postulated can be studying,talking, eating, and drinking. More so, if the given user’s communitymembers only previously talked, ate, and drank, it is with a higherprobability to postulate that the given user is talking, eating, anddrinking rather than studying. Furthermore, if the given user visitedLocation A previously, but only talked and drank, then an even higherprobability is that the user is currently talking and drinking ratherthan eating and studying. It is within the scope of this invention topostulate some or all of the previously detected activities of a givenlocation. More so, it is within the scope of this invention to rankorder the activity suggestions according to the relevance of thepreviously visiting users to the given current user. As previouslydescribed, the system can request confirmation of suggested activitiesthrough the user’s mobile device.

The system can additionally or alternatively reduce possible locationtypes and/or user activities as a function of the past location typeand/or user activity of the positional destination as a function of thetime of day. The system can rank possible location types and/or useractivities of the positional destination based upon known past timeperiods corresponding to the time of day of the current user visit. Forexample, again given Location A, if previous visiting users wererecorded to study one or more times during the intervals: 3:00 - 4:30 PMand 7:30 -9:00 PM, and to drink one or more times during the intervals:4:00 - 7:00 PM and 8:30 PM - 2:00 AM, then a current visiting user atLocation A at 3:15 PM is likely studying, at 4:15 PM is likely to beeither studying or drinking, and at 1:00 AM is likely to be drinking.More so, if the given user’s community members only studied between3:15 - 4:30 PM then it is with a higher probability to postulate thatthe given user is studying rather than drinking at 4:15 PM. Furthermore,if the given user visited Location A previously but only studied, thenan even higher probability is that the user when at Location A isstudying. It is within the scope of this invention to postulate some orall of the previously detected activities of a given location. More so,it is within the scope of this invention to rank order the activitysuggestions according to the relevance of the previously visiting usersto the given current user. As previously described, the system canrequest confirmation of suggested activities through the user’s mobiledevice.

In embodiments of this invention, time context alone can be used topostulate activities. For example, if most days, a user is recorded tobe drinking coffee between 9:00 -10:00 AM, then, without contradictoryinformation, a plausible activity postulate is that at 9:35, the user’sactivity is drinking coffee. Again, as previously disclosed, it iswithin the scope of this invention to rank order the postulated activitysuggestions according to the relevance of the previous users to thegiven current user and/or to obtain confirmation of suggestedactivities.

Additionally, it is also within the scope of this invention to rankorder the time postulates based on frequency of occurrence within thetime interval. This rank ordering applies to both location based andlocation independent time based postulates. For example, if in theinterval 4:00 - 4:30 PM, community members studied 25 times but drank 5times then, at 4:15, it is with a higher probability to postulate thatthe given user is studying rather than drinking.

In embodiments of the MPSM method and system of this invention, thesystem can search and/or use, if available, external, third partyinformation about the positional destination for postulating activitiesfor a given location. For example, third party vendors might provide,either free of charge or for a fee, activity information for a givensite. Consider a marketing website of a centralized homepage for agrocery store chain. Such websites are likely to contain addresses ofmany or all of the associated stores. Since these stores all supportshopping, an activity associated with these locations is shopping.Similar information can be derived or purchased from other sources suchas but not limited to commercial information repositories. Additionally,maps can be parsed. Given a location of a road, an activity of thatlocation is likely to be driving. Various and alternative third partyinformation gathering approaches and their incorporation into activityclassification and postulation can be incorporated into the method andsystem of this invention.

Suggested activity information, particularly but not limited toinformation obtained or derived from third party vendors, might beadditive or might be contradictory. Thus, combining or reconcilingpotential activities is needed. The use of voting schemes, biased basedon credibility of the source or on frequency, such as majority, or otherknown techniques, can be incorporated in the method and system of thisinvention. Note that differing suggested plausible activities mayadditive or may be contradictory. The use of techniques such as, but notlimited to, conflict resolution methods, ontology determination, andclustering, etc., can be incorporated to recognize potential conflictsand to expand classification is within the scope of this invention.

Additionally, the classification of plausible activities based onactivities occurring in the surrounding vicinity is likewise within thescope of this invention. For example, consider an unknown locationadjacent to two known locations, such as, but not limited to, twoneighboring stores or two neighboring beaches. For the neighboringstores, known activities might include shopping and strolling, while forthe neighboring beaches, known activities might include sunbathing andswimming. Given location proximity, it is within the scope of thisinvention to suggest a user’s activity at the unknown location to beeither shopping and strolling or sunbathing and swimming, respectively.Confirmation can always be obtained for suggested activities and to biassuggested activities based on user familiarity and frequency ofoccurrence.

In embodiments of the MPSM method and system of this invention, localusers can opt to delay their notifications. That is, once a location isvisited or an activity occurs, a local user can opt to have thenotification of the location or activity be delayed by some period oftime. Delaying a notification provides the local user with the abilityto notify their community of the location visit or activity occurrence,but still provides the local user time to progress to the next locationor activity. As discussed above, users can also choose to automaticallyshare or not share photos taken with the digital picture frames of thisinvention.

Notifications can be complemented with correlations with other communitymembers. That is, both the local user and their respective community canbe automatically notified with a comparison. A comparison, for examplebut not limited to, can identify other community members havingpreviously conducted a specific activity or having visited a givenlocation previously. Comparisons are made by checking other communitymember locations and activities against those of the local user.Checking is performed via a search of the location-activity database. Ifa match exists within a specified or predetermined period of time, acomparison notification is made automatically. The period of time can bearbitrarily set or can follow some logical time quantum such as hour,day, week, etc.

Locations and activities are known by name. However, in addition to aname, locations and activities can have associated personal labels.Labeling locations and activities can detail familiarity to the locationand activity. User labels for locations can be surrogate names, forexample, “favorite city” for Chicago, can be songs or sound waves, forexample song words “my kind of town, Chicago is” for Chicago, can be apicture, for example “the Water Tower” for Chicago, can be a video, forexample “a panoramic view of the Chicago skyline” for Chicago, or anycombination of these and other multimedia tags supported by the localdevice. Similarly, user labels can exist for activities. For example,“favorite vice” for drinking wine, or it can be a song or sound wave,for example the song words “a bottle of red” for drinking wine, or itcan be a picture, for example, a wine bottle picture for drinking wine,or it can be a video, for example “a panoramic view of a vineyard” fordrinking wine, or any combination of these and other multimedia tagginglabels supported by the local device.

In embodiments of the MPSM method and system of this invention, localusers and community members can comment on their own and each other’slocations and activities. Comments can take any of the many multimediaforms provided by the local device. These include, but are not limitedto, text, sound, pictures, videos, or any combination thereof. Multiplecomments can be made by the local user, their community, or combinationthereof. In addition to stating their opinions (commenting), communitymembers can prompt for clarification. That is, by issuing “what”comments, community members request additional information on the postedlocations and activities. Additionally, user can “like” their own andeach other’s locations and activities. By “liking” a location oractivity, community members express their satisfaction of theirrespective community members’ presence in terms of location andactivity. Multiple community members as well as the local user can“like” a location and activity.

The MPSM method and systems of this invention can track vast data onboth the local user and their respective community members. These datacover, including but not limited to, locations, activities, and alsoindividuals both who are system users and those who are not. These datacan be stored and summarized. A summary of the local user and communitymember locations, activities, time durations involved in each of theselocations and activities, individuals who they encountered, etc., can becomputed and presented to the user. This summarization can range fromsimple statistical aggregation to advanced correlations as derived byany of the many, both individually and combined, machine learning, datamining, business analytics, and statistical techniques known in the art.

Information that can be aggregated or derived can answer, exemplary butnot limiting, questions such as: how much time a local user spent doingthings, such as, working at home, working out, walking the dog,commuting to work?; how much time a particular community member spentdoing things, such as, working at home, working out, walking the dog,commuting to work? (Note that the information derived for the communitymember is based strictly on the information that that particularcommunity member chose to share.); who are the five most commonindividuals that a particular user interacts with?; what is thelikelihood that after seeing a particular user, the given local userwould see a particular different individual?; which activities andlocations are most closely associated with each other and when are theymost likely to occur?; which three users among a given community aremost likely to visit a particular location together?

Local users can be provided with summaries of their locations, durationsat these locations, and activities at these locations. Furthermore, atthe discretion of the local user, these summaries are made available totheir community members.

The system can also generate and maintain both aggregation and derivedinformation. This information can be used to optimize suggestions toavoid obstacles, for example, but not limited to preferred routing ofcommuting path, promoted target advertising, for example but not limitedto location of nearby ice cream store for those users who frequentlyrecord “eating ice cream” as an activity, and a host of otherinformational services known in the art.

The following examples illustrate, without limitation, the abovediscussed data capturing, storing, analyzing, mining, and presentingMPSM functionalities of this invention. It is to be understood that allchanges that come within the spirit of the invention are desired to beprotected and thus the invention is not to be construed as limited bythese examples.

This invention further provides a means to rapidly display and reviewuser photographs. In one embodiment, the device, such as through anexecuted application, includes and executes code instructions forproviding rapid feedback in an ergonomically convenient, intuitive userinterface. The code allows for displaying user photographs or other datain response to a plurality of user hand movements within a continuoususer swipe on a mobile device screen. User hand movements are eithercaptured via a touch screen or via gestures captured via the line ofview of an included camera.

By a user interface method, referred to as the Ownbey-scrub, or “scrub”for short, a user can hold down an icon to select and swipe through aclustered collection of pictures representing activities that occurwithin a given time span, location, with specific other users, or anyother clustering condition. Clustering of items can be accomplished inany of the many known in the art clustering techniques.

In embodiments of this invention, the scrub operates by placing a finger(and holding it) on an icon presented on a touch screen and scrubbing(without lifting off the screen) with that finger horizontally (left orright) or vertically (up or down). In response to the direction of thecontinuous finger contact/movement, quick feedback is provided to theuser in the form of rapidly displayed pictures. Left or right and up ordown represent forwards or backwards depending on user preference. Userpreference might differ due to any orientation (left or right handed) orany physical or logical conditions. By default, right to left and top tobottom is ascending order while left to right and bottom to top isdescending order. Composition movements consisting of both vertical andhorizontal can also be used. For example, a user might push and holdher/his thumb and scrub right to left, without lifting the hold,scrolling rapidly through her/his daily events from morning to night.Determination of direction of scrolling can be accomplished using anyknown in the art vector composition methods. Scaling the tracking of thescrub movement based on the number of pictures in the intended scrolland determining when such movement was performed can be done via anyknown in the art computations as used for such processing in the imageanalysis, graphics, and user interface domains. Additionally, it iswithin the scope of the invention to support voice activated commandsthat support similar scrub operations.

The invention further includes ranking the order the pictures arepresented within each scrub, and/or ranking the order the presentationof the pictograms associated with each user. Rank ordering can be basedon any of the many ordering criteria disclosed herein and/or known inthe art including but not limited to chronological time associated withthe set of activities, location of the activities ordered based on biasof locations, proximity to other users - either a specific user or anyusers within a user’s community, proximity to a location where the lastphotograph was taken, and so forth.

Thus, the invention provides a digital picture frame including a cameraconnected to the frame, and a network connection module for use as adevice for displaying pictures from a user’s electronic device and/orsocial media account or her or his community members’ social mediaaccounts. The frame allows for efficient, automated access to photosrelevant to the viewer(s) of the frame. The automated frame allows forchanging photos for the viewer(s) without multiple manual steps, and foridentifying and/or displaying two or more related photos forsimultaneous split-screen display.

The invention illustratively disclosed herein suitably may be practicedin the absence of any element, part, step, component, or ingredientwhich is not specifically disclosed herein.

While in the foregoing detailed description this invention has beendescribed in relation to certain preferred embodiments thereof, and manydetails have been set forth for purposes of illustration, it will beapparent to those skilled in the art that the invention is susceptibleto additional embodiments and that certain of the details describedherein can be varied considerably without departing from the basicprinciples of the invention.

What is claimed is:
 1. A method of displaying digital photos on adigital picture frame with a digital display screen and a networkconnection module, the method comprising: connecting to a firstinteraction application of a first electronic device of a first userthrough the network connection module, the first electronic deviceincluding a first digital photo collection; connecting to a secondinteraction application of a second electronic device of a second userthrough the network connection module, the second electronic deviceincluding a second digital photo collection; automatically identifyingand selecting two different photos that include a matching photo contentfrom the first and/or the second selected digital photo clusters; anddisplaying the two different photos simultaneously on the digitaldisplay, each in one of two equal sections of the digital display,wherein a first of the two different photos is in a first of the twoequal sections, and a second of the two different photos is in anadjacent second of the two equal sections.
 2. The method of claim 1,wherein the matching photo content is a same person or persons.
 3. Themethod of claim 2, wherein the two different photos are photographicallycaptured in two different calendar years, wherein the two differentcalendar years are separated by more than one additional calendar year.4. The method of claim 3, wherein the first of the two different photosis a baby photo of a person in a first calendar year and the second ofthe two different photos is of the person at another age separated bymore than one additional calendar year; and further comprising: changingthe second of the of the two different photos with a slideshow of theperson at other ages, wherein the baby photo is not changed on the firstof the two equal sections.
 5. The method of claim 3, further comprising:automatically clustering from the first digital photo collection, viaautomated facial recognition, photos over a multi-year timeframe of theperson to obtain clustered photos of the person, wherein the clusteringcomprises: preparing a plurality of sub-clusters each including digitalphotos of the person at a similar time and/or age; extracting photofacial features within each of the plurality of sub-clusters thatcomprise a predetermined limited change over the timeframe to form acluster representation of the person for each of the plurality ofsub-clusters; comparing cluster representations between the plurality ofsub-clusters to determine related sub-clusters of the person over thetimeframe as a function of the cluster representations; linking orgrouping related sub-clusters to form the clustered photos of the personfor display; and automatically selecting a first of the two differentphotos from a first of the sub-clusters and a second of the twodifferent photos from the second of the sub-clusters.
 6. The method ofclaim 1, wherein the matching photo content is a same person and each ofthe two different photos further include one of two different secondpersons, wherein the different second persons are related familymembers.
 7. The method of claim 6, wherein the two different photos areautomatically downloaded from one or more digital photo collections ofthe two different second persons.
 8. The method of claim 6, furthercomprising: automatically clustering digital photos of the first andsecond digital photo collections according to content; automaticallyselecting the two different photos as a function of the clustering. 9.The method of claim 8, wherein the two different photos are selectedfrom two different photo clusters.
 10. The method of claim 1, furthercomprising: automatically scanning digital photos of the first andsecond digital photo collections for content; and automatically creatinga plurality of photo clusters of the digital photos for display from thefirst and second digital photo collections, wherein each of the photoclusters is based upon a different cluster content; and selecting thetwo different photos from at least one of the photo clusters.
 11. Themethod of claim 10, wherein the two different photos are selected from aphoto cluster for the matching photo content.
 12. The method of claim10, further comprising: filtering each of the photo clusters as afunction of quality, wherein photos of a predetermined low quality areremoved from the clusters; and filtering the each of the photo clustersas a function of a further content, wherein photos includingpredetermined people, activities and/or locations are removed from thephoto clusters.
 13. The method of claim 1, wherein the two differentphotos are displayed in a slideshow of digital photos on the digitaldisplay, wherein a plurality of displayed slideshow photos isautomatically downloaded from electronic devices and/or social mediaaccounts of a plurality of community members.
 14. The method of claim 1,wherein the digital display automatically determines an identity of aviewer of the digital display as the first user.
 15. The method of claim14, further comprising: automatically connecting to the firstinteraction application of the first user over a network; andautomatically selecting the two different photos from the first selecteddigital photo cluster.
 16. The method of claim 15, further comprising:automatically identifying preferred photos or restricted photos in thephoto cluster as a function of user information.
 17. The method of claim14, further comprising: detecting an electronic device of the viewer inproximity to the digital display; and automatically scanning andclustering digital photos of the electronic device.
 18. The method ofclaim 1, wherein a first of the two different photos is from the firstselected digital photo and a second of the two different photos is fromthe second selected digital photo cluster.
 19. The method of claim 1,further comprising: automatically clustering photos of the first andsecond digital photo collections over a timeframe for display, whereinthe clustering comprises: preparing a plurality of sub-clusters, eachfor a common detected content in the photos selected from things,places, activities, or combinations thereof, wherein each of theplurality of sub-clusters is formed of the common detected content at asimilar time and/or age; linking related sub-clusters of the commondetected content together to form a cluster of related common detectedcontent for display as a function of one or more predetermined featureshaving limited change over a timeframe; and automatically selecting afirst of the two different photos from a first of the sub-clusters and asecond of the two different photos from the second of the sub-clusters.20. The method of claim 19, further comprising: extracting photofeatures of the common detected content within each of the sub-clustersof photos to form cluster representations, wherein the photo featuresare predetermined as having the limited change over a timeframe;comparing the cluster representations between sub-clusters to determinethe related sub-clusters; and linking or grouping the relatedsub-clusters for display as a function of matching clusterrepresentations.